We receive daily end-of-day data from a data vendor (i.e. not direct from an exchange) and are comfortable with this.
We are now wanting to receive live data, and after a few enquiries we are feeling tempted to go direct to the exchanges rather than to a vendor.
Obviously we expect the work of dealing with live feeds to be different from dealing with CSV files (which is typically what you do for end-of-day data), and we are carrying out our due diligence to see what is involved in receiving/parsing/storing real-time data.
Do you have any suggestions or know of any guides, tutorials, or advice pages on what is involved if you want to receive real-time data feeds?
For example, the CME Group pages (link) seem very thorough, but I am hoping to find something like a 'Real Time Data Feeds for Dummies'.
Are there any significant reasons why you would recommend getting live data from a vendor rather than direct from the exchanges?
- we are focussed on fixed-income products futures & options, and need data from only 2 exchanges (CME Group and ICE),
- we work mostly with Python and R, and have experienced C/C++/C# coders in the team,
- we do not need any GUI front-end applications for browsing the data or for doing analysis, we just want to get the data into our database so that our in-house applications can use it.
Update The reply from @chollida raises an important point about the connectivity: wherever you get your real-time streaming data from you are going to have to demonstrate, to some degree, that you have a properly secure connection and that you have a proper audit on the way you use that data. So let me add another couple of questions:
- For real-time data are the network security/connectivity and data-usage compliance/audit obligations something that we should worry about to the point that we should consider bringing a network expert into the team?
- Are the network security/connectivity and data-usage compliance/audit obligations easier to satisfy if you take your data from a vendor or from the exchange?