Efficient integration of tick data feed with signal generation

Question

The goal is to design the integration of processes generating trading signal doing analytics on a stream of asynchronous tick data retrieved using the native Python TWS API of Interactive Brokers. Two relevant questions here and here, but they focus on data storage only and not on the integration with the analytics part.

I see two major design options:

1. Integrate all processes in a single application. Whenever new data arrive, append the new data in memory to do analytics AND in a file on disk in order to avoid losing all data if the program crashes. PRO: everything is seamlessly integrated and all data are in memory, so no time is wasted in IO operations.

2. Keep all processes separated. Here we have the TWS API receiving asynchronous data feed and append the data in a flat/binary file in the system. Multiple independent analytics applications read-only the flat/binary file and do and output potential trading signals to a .txt that will be read and executed by the TWS API. PRO: modularity. If a error occurs in generating the trading signals, or if we want to stop/change a small component during trading hours, we can just restart the single process and not the whole program.

Question: which design option might be considered more reasonable? Feel free to suggest variations and best practices.

Extra: in option 2, which flat/binary files types could be suitable to support multiple read-only and a single write/append-only processes at the same time without the need to serialize these operations in a single thread in a single process (e.g see caveat here for pandas.HDFStore)?

Answer 1

I've used both approaches at different points of my career, and both are valid.

Firstly, there's a few statements that aren't quite correct.

...AND in a file on disk in order to avoid losing all data if the program crashes.

... have the TWS API receiving asynchronous data feed and append the data in a flat/binary file in the system. Multiple

It sounds to me that you plan to write to disk on the first step of your entire system to achieve persistence. This still doesn't guarantee anything - your disk could fail, your RAID array could fail then your disk could fail during resilvering due to the bit error rate on large disks, your OS could be buffering the write in page cache and your machine crashes before it is flushed. In fact, given the stability of servers nowadays, your disk is probably more likely to fail than memory.

If you write to disk in the hot path every other optimization you're thinking of is probably meaningless.

PRO: everything is seamlessly integrated and all data are in memory, so no time is wasted in IO operations.

Having a single application run as a single process, a single application run as multiple processes, or multiple applications (each run as a separate process) has no bearing on whether your data is in memory or disk. Practically all multiprocess applications for trading will employ shared memory or domain sockets to communicate.

which flat/binary files types could be suitable to support multiple read-only and a single write/append-only processes at the same time without the need to serialize these operations in a single thread in a single process (e.g see caveat here for pandas.HDFStore)?

Generally there's no thread safety issues with a single writer and multiple readers to the same file, and your file serialization format does not affect this. If your readers have independent state, you can use any format you want. If you need to coordinate state though, for example reader 1 reads as many bytes until it fills its read buffer then does some work with that, and reader 2 needs to follow up sequentially where reader 2 left off, then you end up with race conditions.

Most concurrency issues with file I/O arise in multiple writer implementations.

So you can get away by setting things up with a single writer and multiple readers with independent state.

---

Multiprocessing

Pros:

• Easier to implement since each application directly mapped onto an OS process, and you can rely on the OS for scheduling and to isolate memory overflow.
• Easier to profile and use debugging tools against applications modularized across processes than threads.

Cons:

• Processes need to share memory with separate address spaces, relies on OS support for shared memory access.
• Less memory efficient than threads since processes maintain more state (e.g. switching context, network handle table).

Multithreading

Pros:

• Does not rely on OS shared memory constructs for communicating between pieces.
• More scalable due to smaller memory footprint of threads.

Cons:

• No OS protection against separate threads from overflowing onto each other's memory.
• Difficult to support multiple platforms due to different threading interfaces.