Real-time market data from the exchanges: what should we be aware of?

Question

We receive daily end-of-day data from a data vendor (i.e. not direct from an exchange) and are comfortable with this.

We are now wanting to receive live data, and after a few enquiries we are feeling tempted to go direct to the exchanges rather than to a vendor.

Obviously we expect the work of dealing with live feeds to be different from dealing with CSV files (which is typically what you do for end-of-day data), and we are carrying out our due diligence to see what is involved in receiving/parsing/storing real-time data.

Questions:

• Do you have any suggestions or know of any guides, tutorials, or advice pages on what is involved if you want to receive real-time data feeds?
  For example, the CME Group pages (link) seem very thorough, but I am hoping to find something like a 'Real Time Data Feeds for Dummies'.

• Are there any significant reasons why you would recommend getting live data from a vendor rather than direct from the exchanges?

FYI:

• we are focussed on fixed-income products futures & options, and need data from only 2 exchanges (CME Group and ICE),
• we work mostly with Python and R, and have experienced C/C++/C# coders in the team,
• we do not need any GUI front-end applications for browsing the data or for doing analysis, we just want to get the data into our database so that our in-house applications can use it.

Update The reply from @chollida raises an important point about the connectivity: wherever you get your real-time streaming data from you are going to have to demonstrate, to some degree, that you have a properly secure connection and that you have a proper audit on the way you use that data. So let me add another couple of questions:

• For real-time data are the network security/connectivity and data-usage compliance/audit obligations something that we should worry about to the point that we should consider bringing a network expert into the team?
• Are the network security/connectivity and data-usage compliance/audit obligations easier to satisfy if you take your data from a vendor or from the exchange?

Answer 1

Do you have any suggestions or know of any guides, tutorials, or advice pages on what is involved if you want to receive real-time data feeds?

You won't find a how to for consuming data feeds direct from the exchanges for several reasons

1. Each exchange is different, from format to api
2. Its hard to write a tutorial as the data is owned by the exchanges and typically can't leave your firm, as spelled out in your market data agreement.
3. Each fund typically writes their own code to parse the data feeds and like most funds keep it proprietary, so there isn't one unified library that you can use.
4. One big issue is how you are going to receive the data. Typically you'd be collocated with each exchange you want data from.

How are you planning on connecting to the exchange? colocate at each exchange? software vpn to each exchange?

The other big issue is that typically the exchanges want a guarantee that the data is only used for hte purpose you say it is. What software are you using for tracking hte data and limiting it to only qualified computers or individuals? The exchanges will want to know.

Are there any significant reasons why you would recommend getting live data from a vendor rather than direct from the exchanges?

Yes, the big one is that you can get dat from multiple exchanges this way, ie you want NYSE, NASDAQ and CBOE, a vendor can get data and merge them all together for you. They also do the compliance for you,

If you don't need the fastest possible quotes then a vendor makes things 100 times easier on you.