I'd like ask everyone a more concurrency programming but definitely quant-finance related question. How do you deal with staleness of data in market hours as quote ticks are streaming and your model is calculating its values based on previous ticks (possibly stale quotes) to make trading decisions?
Concrete example, deriving the implied volatility of an ETF according to the IV of its components. In an ETF with several hundred components, quote ticks are streaming in constantly and the model needs to respond to the changing ticks. Let's assume that there is no way to derive a composite IV with a single quote tick but the model has to be re-calculated as a whole.
The naive concurrent programming model of lock on write would make the app woefully inefficient as there'd be thousands of writes (quote ticks) before the app can "read" from it.
The other naive approach is to make the updated quote ticks visible to your reader thread, the app thread or business logic. However, this gets complicated because it forces the algo writer to be conscious of writing good concurrent code and know the concept of atomicity and immutability (e.g., what happens if you derive a metric half way in your code based on stale quotes before it got stale, and then use that variable to make the trading decision).
I'm curious how you have solved or thought about this problem. Thanks in advance for your help and much appreciated!
