Analysing the MathLink protocol

All 'lines' appearing below are terminated by \r.

Note that the last byte before the 'string part' is just telling us the length of the string part -- e.g. in 'FAYJCallPacket', J=10  is (correctly!) the length of 'CallPacket'. FAYJCallPacket FAYNFrontEnd`Value SN$VersionNumber FA#OA~LReturnPacket "IC~C.5...2 ]] Let's have a look at . Not all the '.'s are really dots here; half of them are actually hex 00.

F ('FullForm')
The second byte tells us the number of leaves. After that, there's either a single mysterious byte, or 4 mysterious bytes (beginning with #?). After that there's the string length specification.

FY or F#A~

Here  tells us the number of subsequent leaves,  is the length of and  is the 3+.

S
Means 'String literal'?

Always followed by a capital letter or a decimal number and a space. These indicate the length of the rest of the line.

Y
seems like 'S'. Line starts with a Y, then a single letter (indicating a length), then a string, which allows seems to be a 'SomethingForm'.

]
some sort of packet terminator? sometimes 1, sometimes 2?

Strings, and a length specification
FBYMMessagePacket YEPower SDinfy FAYPExpressionPacket FBYGBoxData FAYFRowBox FCYDList FAYFRowBox FCYDList SEPower SB:: SF"infy" SB: S159 "Infinite expression \!\(1\/0\) encountered. \!\(\*ButtonBox[\"More. &\", ButtonStyle->\"RefGuideLinkText\", ButtonFrame->None, ButtonData:>\"Power::infy\"]\)" YLStandardForm FAYPOutputNamePacket SHOut[5]= FAYVReturnExpressionPacket FBYGBoxData SOComplexInfinity YLStandardForm FAYOInputNamePacket SHIn[6]:=

Okay... look in the middle of this packet, at the lines SEPower SB:: SF"infy" SB: S159 "Infinite expression \!\(1\/0\) encountered. \!\(\*ButtonBox[\"More. &\", ButtonStyle->\"RefGuideLinkText\", ButtonFrame->None,ButtonData:>\"Power::infy\"]\)"

The second bytes of the first four lines are telling you the length of the rest of the line; when you see Sk, where k is the k-th letter of the alphabet, then there are k bytes following. In the last line, there are exactly 159 characters after 'S159 '.