I spent some time trying to design some parts of the system, going over the information I found.
At the top-level, the system looks like this
- A "real-time" module that receives market data, processes it, creates a local limit order books and some additional information, there are some "bots" that implement various strategies, and a part that is responsible for creating / updating / canceling orders.
- There is also an "analytical" module, the global goal of which is the creation and use of predictive models, whose predictions are used by some strategies. This part of the system should be able to work with historical data (for the last day, month, etc.)
I have a few questions that are more related to details.
- There are many different design patterns for Event Driven systems. Which of them is preferable for this task?
- What's the best way to store data in a "real-time" module? For example, a local limit order books. Could it just be a data structure in memory? If yes, how can we avoid multiple locks? Can an in-memory database be used, or would it be expensive?
- You also need the ability to access historical data to create predictive models. If I understand correctly, we must first choose an OLTP storage (are there any best practices on this?)
- In this case, the market data flow will, on the one hand, pass through the "real-time" processing module to create / update local limit order books and, at the same time, be stored in some storage with next processing and normalization in OLAP storage ?