NYSE binary data, convert to ASCII

Question

The data product "TAQ NYSE Order Imbalances" from the New York Stock Exchange is in a format that is described pretty well in sections 4.8, 4.9, 4.10, and 5 of the document "NYSE Order Imbalances Client Specification", version 1.12, q.v. Briefly, it's a mix of ASCII and binary: stock symbols, for example, are in plain text, but padded by null bytes, while numerical fields are in binary digits represented by a byte with that binary value. All fields are fixed-width, so data rows simply follow one another.

Does anyone know how to convert this to, say, a comma-separated file?

Answer 1

I think you do not need to be a "Systems Programmer", certainly not an experienced one, to solve this problem:

1) Focus on the header, its your legend to the file structure. It describes the format and essentially already tells you how to decode the following messages.

2) Depending on your choice of language you then process each message in binary format and convert each item to the numeric format. In C#, some use "BitConverter" but obviously C# is not the language of choice here. If you can tell me which specific language you use to make the conversion then that would be helpful. A lot of people use Python to convert this kind of stuff to a higher level text based format such as csv or any delimiter-delimited structure.

3) Before you convert you may want to think carefully whether you may want to perform operations on the byte array representations of your numeric values (I am not familiar with your mentioned specific feed, though some feeds only output the "alpha" rather than full spread, for example, thus you need to perform add/subtract operations which can in certain cases be more optimal to perform on the byte array itself). Here is an example : http://stackoverflow.com/questions/3641274/c-sharp-int-byte-conversion

Here are couple Python examples just to show you how a simple byte[]-> int conversion could be done:

http://stackoverflow.com/questions/386753/how-do-i-convert-part-of-a-python-tuple-byte-array-into-an-integer

http://stackoverflow.com/questions/444591/convert-a-string-of-bytes-into-an-int-python

P.S.: It won't help you but I find mixed message formats very inefficient but that is not your fault. Most efficient streams only ship byte arrays, nothing else. A symbol should anyway never be in string format internally, but rather be assigned an int32 or int64 code. Mapping internally is much faster than converting each symbol of each message from byte array to string. Also, even if the symbol is decoded in ASCII that is very inefficient and blows up message size.

Answer 2

I wrote the pack R package (based on Perl's pack function), to do this for opentick (now defunct) data. You can look at the opentick package (in the CRAN archives) to see how I used it.

I just noticed that you said you're comfortable with Perl in your SO post. In that case, I'd recommend you use Perl's unpack function.