3
$\begingroup$

I have designed a fast limit order book, similar to the one described here. It is working and passing the unit tests. Now, I am thinking of its reliability. i.e in case of failure (e.g unhandled exception or machine restart etc)

I have sketched a design, that after every hour, we store the whole state of the orderbook in a Database. Then in case of failure, we can recover the last hour's state and continue it. In a order matching engine with >>1000 Transactions/Second (TPS) , is this even feasible to use a Database for disaster recovery of an orderbook?

$\endgroup$
1

1 Answer 1

6
$\begingroup$

For data on a given single host, a common architecture is to write the data onto a queue (could be multi-consumer or single consumer), and have an asynchronous writer that reads it off and persists it in parallel to the critical path operation.

Beyond that, it doesn't quite matter how you persist your order book for recovery. There's many "OK" ways to do it and the optimal design will depend a lot on other business requirements.

It is possible to accommodate 1000 transactions per second and more with today's hardware on a typical DBMS. But that said, committing event-by-event to the database wouldn't be a very conventional or efficient pattern. If you intend to store them in a database ultimately, you will likely want to buffer up the events somehow and commit them in batch. You could do that in-memory with redundant hosts; with a flat file on host; with a more modern, likely distributed, database management system designed for high rate of inserts per second; or with a messaging framework that has persistence like Kafka - really anything.

If you really want to do it correctly though, I would recommend looking into hardware-driven solutions such as an optical tap; port mirroring; a layer 1 switch; a dedicated capture card, or commercial messaging solutions such as Solace, Informatica or Tibco. Persist the incoming order messages as soon as they arrive on the edge of your network, even before you update the order book.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.