1
$\begingroup$

We receive daily end-of-day data from a data vendor (i.e. not direct from an exchange) and are comfortable with this.

We are now wanting to receive live data, and after a few enquiries we are feeling tempted to go direct to the exchanges rather than to a vendor.

Obviously we expect the work of dealing with live feeds to be different from dealing with CSV files (which is typically what you do for end-of-day data), and we are carrying out our due diligence to see what is involved in receiving/parsing/storing real-time data.

Questions:

  • Do you have any suggestions or know of any guides, tutorials, or advice pages on what is involved if you want to receive real-time data feeds?
    For example, the CME Group pages (link) seem very thorough, but I am hoping to find something like a 'Real Time Data Feeds for Dummies'.

  • Are there any significant reasons why you would recommend getting live data from a vendor rather than direct from the exchanges?

FYI:

  • we are focussed on fixed-income products futures & options, and need data from only 2 exchanges (CME Group and ICE),
  • we work mostly with Python and R, and have experienced C/C++/C# coders in the team,
  • we do not need any GUI front-end applications for browsing the data or for doing analysis, we just want to get the data into our database so that our in-house applications can use it.

Update The reply from @chollida raises an important point about the connectivity: wherever you get your real-time streaming data from you are going to have to demonstrate, to some degree, that you have a properly secure connection and that you have a proper audit on the way you use that data. So let me add another couple of questions:

  • For real-time data are the network security/connectivity and data-usage compliance/audit obligations something that we should worry about to the point that we should consider bringing a network expert into the team?
  • Are the network security/connectivity and data-usage compliance/audit obligations easier to satisfy if you take your data from a vendor or from the exchange?
$\endgroup$
  • $\begingroup$ Here is an overview post I found: codeproject.com/Articles/553206/…. It also discusses an API that the author has developed. Here is an article from someone at TIBCO on solutions for handling streaming data : infoq.com/articles/stream-processing-hadoop $\endgroup$ – Robert Jan 21 '15 at 10:27
  • $\begingroup$ Make sure that your database infrastructure can ingest the market data at the necessary rates. $\endgroup$ – sevzas Jan 24 '15 at 21:10
1
$\begingroup$

Do you have any suggestions or know of any guides, tutorials, or advice pages on what is involved if you want to receive real-time data feeds?

You won't find a how to for consuming data feeds direct from the exchanges for several reasons

  1. Each exchange is different, from format to api
  2. Its hard to write a tutorial as the data is owned by the exchanges and typically can't leave your firm, as spelled out in your market data agreement.
  3. Each fund typically writes their own code to parse the data feeds and like most funds keep it proprietary, so there isn't one unified library that you can use.
  4. One big issue is how you are going to receive the data. Typically you'd be collocated with each exchange you want data from.

How are you planning on connecting to the exchange? colocate at each exchange? software vpn to each exchange?

The other big issue is that typically the exchanges want a guarantee that the data is only used for hte purpose you say it is. What software are you using for tracking hte data and limiting it to only qualified computers or individuals? The exchanges will want to know.

Are there any significant reasons why you would recommend getting live data from a vendor rather than direct from the exchanges?

Yes, the big one is that you can get dat from multiple exchanges this way, ie you want NYSE, NASDAQ and CBOE, a vendor can get data and merge them all together for you. They also do the compliance for you,

If you don't need the fastest possible quotes then a vendor makes things 100 times easier on you.

$\endgroup$
  • $\begingroup$ After further research it seems to me that your point (3) is not our main worry, because we will only connect to 2 exchanges and therefore may at worst have to code up 2 parsers. The main issue does instead seem to be your point (4) and your comment on guarantees of data auditing: we would not be co-locating and I think we would need to have a decent amount of network expertise in our team in order to handle the technology that supports a guaranteed-secure connect to the exchange with proper auditing of data usage. Anyone know if this something that the Open Source world has already solved? $\endgroup$ – Robert Jan 21 '15 at 10:35
  • $\begingroup$ @Robert, There won't be much in hte way of open source as everything is specific to the exchanges and the parsers change over time. You can get the Nasdaq's ITCH/OUCH spec from their site if you wan't to look at a proprietary format. Other exchanges can provide quotes via FIX, which while slower is often easier to interact with via quickfix/x. Having said that, if you don't need to collocate, just get data from a vendor like Bloomberg, its much easier as they normalize the data for you. $\endgroup$ – chollida Jan 21 '15 at 15:39
  • 1
    $\begingroup$ I want to add that two more benefits of using a vendor are that a lot of vendors will provide a software layer that will (1) normalize all of the feeds to that they look "similar" to your software and (2) keep up with changes to the Exchanges' raw protocols and networking to insulate you from having to change your software/infrastructure. $\endgroup$ – sevzas Jan 24 '15 at 21:08
0
$\begingroup$

I believe the important issue is not how you are going to receive the market data from N different exchanges but how you are going to normalize and generalize them into a single protocol that can be consumed internally by your applications.

Take a look on CoralMD that has an article about how to build a market data infrastructure internally.

Disclaimer: I am one of the developers of CoralMD.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.