I'm an individual trying to build a trading system which will ideally be eventually scalable to 1-15 second resolution intraday trading strategies. I'm having some trouble understanding the difference between data feeds applied to a backtest and data feeds applied to live trading and I have a few specific questions:
Are backtests and live trading typically built upon the same abstraction or operate on the same trading/processing engine?
My first thought is that they are, since they should ideally operate identically for backtesting accuracy and realism
Do live data feeds and historic datasets typically provide an identical interface in their respective handlers for querying/retrieving their datasets?
The current state of my system only runs on historic data for backtests with the trading engine updating the time step of the data handler itself. This doesn't seem like a viable solution for higher frequency real time data since it relies on the trading engine to update the data feed rather than the data feed handler. After some research, it seems like a query-based mechanism would be more suitable for real time data since it gives data management control to the handler. Though I'm having trouble understanding how static historical data could be loaded into data stream to be processed in an identical way.
How are data streams most effectively processed?
Under the assumption that both historical data and live data are fed into event streams. I'm having trouble understanding how those streams are then made to be queried or otherwise predictably retrievable by the trading engine. A time series database seems to make the most sense purely due to its ability to handle the volume of data and store enough lookback data, but most time series databases though are fairly expensive for an individual, and I'm not sure it's the most cost effective way of processing the volume of data. What other options are there for providing an effective query engine for both historical and live data feeds?
I'm probably a bit out of my depth here since I'm still fairly new to this, so please tell me if I'm taking the wrong approach to this. I'd also love some more info as to how these systems are designed, or any other resources/books to read regarding data handling methods.