1
$\begingroup$

As practice, I have been wanting to parse exchange data and try to build an order book algorithm on my own. I found some sample data from NYSE: ftp://ftp.nyse.com/Real%20Time%20Data%20Samples/NYSE%20XDP/.

I tried using tcpdump to read the pcap data. However, that does not get the underlying market data. Does anyone have suggestions on how I could parse the pcap file to read the market data? I am currently hoping to do this in C++ as most high frequency trading places use C++.

Any suggestions and recommendations will be very helpful. Thank you.

$\endgroup$
2
$\begingroup$

I've created an example for how to access UDP packets in a pcap file.

The gist is that you have to skip the Ethernet / IP / UDP headers to reach the payload. That's what gets passed to your feed handler.

As for tcpdump, it won't pass the payload to you, but it's still helpful for verifying that you understand the contents when parsing. Eg.,

tcpdump -r NYSE_XDP_IMB_2.2.pcap -e
$\endgroup$
4
  • $\begingroup$ This is great. Thank you so much. Out of curiosity, when I try running tcpdump -r NYSE_XDP_IMB_2.2.pcap -e; an example of the output is 06:58:33.242922 00:0f:53:43:cd:20 (oui Unknown) > 01:00:5e:00:4d:20 (oui Unknown), ethertype IPv4 (0x0800), length 164: 162.69.165.11.63588 > 224.0.77.32.22032: UDP, length 122. What should I be able to understand from the output? $\endgroup$
    – vpy
    Jun 21 '20 at 13:14
  • $\begingroup$ @vpy That tcpdump is the headers, specifically the Ethernet MAC addresses, IP addresses, and UDP ports for destination and source. Extracting the payload means skipping the headers, but that's hard because the headers can contain all kinds of unexpected items, like 802.1Q tags. So knowing what tcpdump finds means you have some way to spot-check your work. $\endgroup$ Jun 21 '20 at 20:18
  • $\begingroup$ This is probably a stupid question but once we get to the payload, can I convert the payload to human readable format? E.g. would it be possible to cast the payload to a type that I can print out? $\endgroup$
    – vpy
    Jun 21 '20 at 22:37
  • $\begingroup$ @vpy You are what converts the payload, unfortunately. You'll have to manually write a feed handler to consume each bit of data. I wrote a quick example that might give you some ideas. $\endgroup$ Jun 22 '20 at 0:21
3
$\begingroup$

I recommend @chrisaycock's answer for completeness. However if you want a quick and dirty way of extracting the payload, you'd use tshark instead of tcpdump:

tshark -r NYSE_XDP_IMB_2.2.pcap -T fields -e data

This can be useful sometimes because many exchanges (NASDAQ, Australia and SIX Swiss come to mind) typically send you historical samples with only the payload.

For instance if you want to transcode your directory full of NASDAQ pcaps into their sample binary format, parallelized across 40 cores, you just need a magic one-liner:

find . -maxdepth 1 -name "*.pcap" | xargs -I {} -P40 sh -c 'tshark -r {} -T fields -e data | cut -c 41- | xxd -r -p > `basename {} .pcap`.bin'
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.