Need advice about distributed backtesting architecture [closed]

Question

We are working under complex enough distributed trading system where several components will run on different physical machines.

Unfortunately, I'm stuck on part backtesting part. Originally we was planning to use tick as synchronization marker. Idea was working till we not added more complicated interaction logic when components started to interact with each other with back loop.

I'm sure that this problem was solved many times before and dont want to reinvent the wheel... Can anybody share at least basic information about topic?

Answer 1

In database-speak, this problem has been solved with things like two-phased commit. That's used in transaction (money) processing when the machines are separate and nothing can be lost. That design required a unique token (like your Tick would have) but generic (not a tick but your own creation). The process is one machine says "I'm ready to do 'A' if you are." Machine B responds with an acknowledgment on that request token. Then an acceptance from A to B defines a Complete. If no answer from A or B along the way, the transaction is incomplete. A retry can happen. The medium for making and sending tokens/requests is up to you. This is the conceptual design and helped make Oracle a fortune 500 company. With JSON and various other post-SOAP era (REST and all the variations) web-service contracts, you can build a good infrastructure. The development language and tools will make it easy or hard for you. I made an asset/machine-tracking system (with Satellite and Cellular data) all built on similar concept. It supported REST and other formats for sending tokens (from my server to a satellite that talks to a fire-fighting pump). If it's a long chain, you need to build that into your design. You need to know what formats are super fast and survive interruptions at the proper layer: eg TCP vs UDP. Sounds like a fun project.