SecuredTransfer
The problem comes with a binary that is assumed to be a wrapper for ftp or something similar.
$ ./securetransfer help
Usage ./securetransfer [<ip> <file>]
Running strings on the binary shows the following text of interest:
dH3<%(
dH3<%(
dH34%(
[]A\A]A^A_
someinitialvalue
ERROR: Socket creation failed
ERROR: Invalid input address '%s'
ERROR: Connection failed
ERROR: Can't open the file '%s'
ERROR: File too small
ERROR: File too large
ERROR: Failed reading the file
File send...
ERROR: Socket bind failed
ERROR: Listen failed
ERROR: Accept failed
ERROR: Reading secret length
ERROR: File send doesn't match length
File Received...
Sending File: %s to %s
Receiving File
Usage ./securetransfer [<ip> <file>]
Upon experimentation with the binary, it was determined that running it with no arguments sets up a listener to receive a file, and running it with two arguments sends the contents of a file to the specified IP address.
# This was the listener shell
$ ./securetransfer
Receiving File
File Received...
this is an example file
# This was the sender shell
./securetransfer 127.0.0.1 temp.txt
Sending File: temp.txt to 127.0.0.1
File send...
The PCAP file has two packets with data, #4 and #5. #4 has 2000000000000000 (8 bytes) and #5 has 5f558867993dccc99879f7ca39c5e406972f84a3a9dd5d48972421ff375cb18c (32 bytes). The first packet may be a dud to detect if the connection is open to being made, and the second packet may be the actual data. The first packet may also be some form of key to break the second packet's encryption, but that is not clear yet.
In order to reverse engineer the method of encoding, I used the example above while running a packet capture with Wireshark on all interfaces. Two packets of exactly the same format were captured: bingo. To find them, use the filter tcp && data. It's strange seeing these packets with data, usually only UDP packets have the data field populated, so that may be a clue.
First packet had 2000000000000000 again, but the contents of the second packet were be28ffb71364b3d274bba58a93e09c21f14442c06fe82e5d33446e88fe4f7f8f (32 bytes) this time.
We know that the second packet is hex due to the limited amount of characters used, whether or not the first packet is as well is unknown. One way to come at this is a known-plaintext attack, where we know the plaintext and ciphertext, and try to reverse engineer the encryption method.