Goal : Using deep learning to build a ML model which would predict the right places where a stock price will increase, decrease or stay stable.

For the current question, assume the labels are well defined.

As features I used the limit order book, but I did not get a lot of success. So I realised that I didn't have all the features to accomplish my goal. In reading the following paper https://arxiv.org/pdf/1710.03870.pdf on page 8, I didn't consider the market and cancellation orders. A setback I got is I did not find a vendor to provide the market orders separately from the limit orders. So in the moment I have the full market depth from algoseek. You can find a sample here.

Does the full market depth contain the market orders? If so, can I build just a book containing the limit and market orders I can use for the training phase of the ML model? Do we really need to separate the limit and market orders to feed the ML model.

enter image description here

Please let me know if you question is unclear.

  • 1
    $\begingroup$ Market orders and cancellations would be available by the exchange's tape or by a data provider who records and aggregates the messages. It seems you don't fully understand how market and limit orders work, though. $\endgroup$ – Kch Aug 9 '18 at 2:53
  • $\begingroup$ tr8dr.wordpress.com/2011/03/04/… $\endgroup$ – Alexey Golyshev Aug 9 '18 at 10:27

You are right that there are limit and market orders and nothing else. But I'll clarify it. Market orders do not stay in the order book because they instantaneously matched against limit orders and generate trades. Limit orders depending on their limit price can also be instantly matched fully or partially. So, do you refer to aggressive orders in general (limit or market) or orders that are Market by type?

In any case, each exchange has its own way / data format to inform about it. Plus, data vendors (e.g. algoseek) may also post-process the data and convert it to another format. In this case it's necessary because algoseek needs to provide the data of the same symbols from different exchanges in a uniform format. As you can see below they don't do it very effectively, but it's up to them.

The documentation of the sample data seems to be outdated, but it's easy to guess the format from the content. So, in order to get Market orders or aggressive limit orders from the data or you need to analyse the trades. Here is an example. To make it more compact I removed the columns of Exchange, MMID, and Symbol.

enter image description here

If you wish to see actual Market orders in the data when they arrive, check GDAX data, see the description of the full channel in the API. Here is a short sample of the raw data. Each line starts with a timestamp (set by recording device).

Also, I highly recommend reading this crash course on Market Mechanics to clarify all the above and much more.

To summarize, any trade that you see in the market data is caused either by a Market order or by an aggressive (crossing the spread) Limit order. Basically, you can model a Market order as an aggressive Limit order with limit price deep in the opposite side.

In some cases like with GDAX data you can see explicitly the type of the order that caused a trade, in some cases you can't see it. And in some cases you can back-engineer they type of the order based on the sequence of data updates. It allows to distinguish partially executed limit orders from market orders and fully executed limit orders. See case #4 from top on this diagram from the Market Mechanics article:

enter image description here

  • $\begingroup$ Hello Serg. Your answer is very nice and answer my question, but I am still a bit confused quant.stackexchange.com/questions/41143/…. Do you agree with that? $\endgroup$ – user1050421 Aug 9 '18 at 14:54
  • 1
    $\begingroup$ It talks about some specific data format. What's important is this: any trade is always caused by either a Market order or by an aggressive (crossing the spread) Limit order. See the update in the answer $\endgroup$ – Serg Aug 9 '18 at 16:54
  • $\begingroup$ You proposed GDAX, but all I need is historical data for ML purpose. I can't see that on their website. Are you up to communicate out of quant.SE? I would be interested to make some work with you ;) $\endgroup$ – user1050421 Aug 9 '18 at 16:59
  • $\begingroup$ Sure, write to sergey(at)bookmap.com. I don't propose using their data. This is simply what I know about it $\endgroup$ – Serg Aug 9 '18 at 17:18
  • $\begingroup$ Ok, you work for bookmap. I sent you an email. I would be interested to communicate with you via hangouts or skype. $\endgroup$ – user1050421 Aug 9 '18 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.