What techniques are used for testing order book implementations?

Question

I am finishing the implementation of a limit order book for modeling NASDAQ. The order book works off of the ITCH feed. My question is what techniques are typically used for testing order books. I am considering two methods primarily.

1. Build a canned set of messages to test the functionality and test possible edge cases.
2. Push large amounts of data through it and see if anything breaks.

I am also curious if there are any thoughts on validating the state of the book at any moment in time. I was thinking of using the L1 feed to at least validate that the top of book quotes are the correct prices and sizes.

Note: Please don't suggest things like unit testing. I am specifically interested in techniques specific to order books.

Answer 1

Validating tops against the consolidated is a good method. Obviously the time stamps won't match up, but the event stream should. Bear in mind that this won't tell you much about whether you're getting the inner dynamics of the book correct (for example, did the newly inserted order go into the right spot within a given price level).

You should build canned test scenarios for the basics of the book. For example, if you're order book reports trades, make sure the trade price is correctly reported by an Order Execute At is received (i.e., the price in the message should be reported, not the price of the resting order).

You should also validate your handling of data feed gaps. What happens if you miss a Modify, or a Delete, or an Order Ex. How do you detect this. Sequence numbers can tell you that you missed something, but you don't know what you missed. Inject known errors into your data stream and verify your book manages it properly. Look for crossed/locked markets that result of missed messages and stale order state, inspect by hand and see if you did the right thing.