The goal is to design the integration of processes generating trading signal doing analytics on a stream of asynchronous tick data retrieved using the native Python TWS API of Interactive Brokers. Two relevant questions here and here, but they focus on data storage only and not on the integration with the analytics part.
I see two major design options:
Integrate all processes in a single application. Whenever new data arrive, append the new data in memory to do analytics AND in a file on disk in order to avoid losing all data if the program crashes. PRO: everything is seamlessly integrated and all data are in memory, so no time is wasted in IO operations.
Keep all processes separated. Here we have the TWS API receiving asynchronous data feed and append the data in a flat/binary file in the system. Multiple independent analytics applications read-only the flat/binary file and do and output potential trading signals to a .txt that will be read and executed by the TWS API. PRO: modularity. If a error occurs in generating the trading signals, or if we want to stop/change a small component during trading hours, we can just restart the single process and not the whole program.
Question: which design option might be considered more reasonable? Feel free to suggest variations and best practices.
Extra: in option 2, which flat/binary files types could be suitable to support multiple read-only and a single write/append-only processes at the same time without the need to serialize these operations in a single thread in a single process (e.g see caveat here for pandas.HDFStore)?