1
$\begingroup$

As practice, I have been wanting to parse exchange data and try to build an order book algorithm on my own. I found some sample data from NYSE: ftp://ftp.nyse.com/Real%20Time%20Data%20Samples/NYSE%20XDP/.

I tried using tcpdump to read the pcap data. However, that does not get the underlying market data. Does anyone have suggestions on how I could parse the pcap file to read the market data? I am currently hoping to do this in C++ as most high frequency trading places use C++.

Any suggestions and recommendations will be very helpful. Thank you.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I've created an example for how to access UDP packets in a pcap file.

The gist is that you have to skip the Ethernet / IP / UDP headers to reach the payload. That's what gets passed to your feed handler.

As for tcpdump, it won't pass the payload to you, but it's still helpful for verifying that you understand the contents when parsing. Eg.,

tcpdump -r NYSE_XDP_IMB_2.2.pcap -e
Improve this answer
$\endgroup$
4
  • $\begingroup$ This is great. Thank you so much. Out of curiosity, when I try running tcpdump -r NYSE_XDP_IMB_2.2.pcap -e; an example of the output is 06:58:33.242922 00:0f:53:43:cd:20 (oui Unknown) > 01:00:5e:00:4d:20 (oui Unknown), ethertype IPv4 (0x0800), length 164: 162.69.165.11.63588 > 224.0.77.32.22032: UDP, length 122. What should I be able to understand from the output? $\endgroup$
    – vpy
    Jun 21, 2020 at 13:14
  • $\begingroup$ @vpy That tcpdump is the headers, specifically the Ethernet MAC addresses, IP addresses, and UDP ports for destination and source. Extracting the payload means skipping the headers, but that's hard because the headers can contain all kinds of unexpected items, like 802.1Q tags. So knowing what tcpdump finds means you have some way to spot-check your work. $\endgroup$
    – chrisaycock
    Jun 21, 2020 at 20:18
  • $\begingroup$ This is probably a stupid question but once we get to the payload, can I convert the payload to human readable format? E.g. would it be possible to cast the payload to a type that I can print out? $\endgroup$
    – vpy
    Jun 21, 2020 at 22:37
  • $\begingroup$ @vpy You are what converts the payload, unfortunately. You'll have to manually write a feed handler to consume each bit of data. I wrote a quick example that might give you some ideas. $\endgroup$
    – chrisaycock
    Jun 22, 2020 at 0:21
3
$\begingroup$

I recommend @chrisaycock's answer for completeness. However if you want a quick and dirty way of extracting the payload, you'd use tshark instead of tcpdump:

tshark -r NYSE_XDP_IMB_2.2.pcap -T fields -e data

This can be useful sometimes because many exchanges (NASDAQ, Australia and SIX Swiss come to mind) typically send you historical samples with only the payload.

For instance if you want to transcode your directory full of NASDAQ pcaps into their sample binary format, parallelized across 40 cores, you just need a magic one-liner:

find . -maxdepth 1 -name "*.pcap" | xargs -I {} -P40 sh -c 'tshark -r {} -T fields -e data | cut -c 41- | xxd -r -p > `basename {} .pcap`.bin'
Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.