Recently, I read a great paper by De Prado et al. on backtest overfitting problem in Quantitative Finance titled Pseudo-Mathematics and Financial Charlatanism: the Effects of Backtest Overfitting on Out-of-sample Perfomance.
In the first chapter, they define in-sample (IS) and out-of-sample performance (OOS) as follows:
With regards to the measured performance of a backtested strategy, we have to distinguish between two very different readings: in-sample (IS) and out-of-sample (OOS). The IS performance is the one simulated over the sample used in the design of the strategy (also known as "learning period" or "training set" in the machine learning literature). The OOS performance is simulated over a sample not used in the design of the strategy (a.k.a. "testing set"). A backtest is realistic when the IS performance is consistent with the OOS performance.
The definitions above are pretty straight-forward, however what confused me is the message in the paper that most people look at IS performance of backtesting when evaluating different strategies. Is that really the case in finance?
For example, most of the time when I did backtest in the past I used the so called rolling-window approach: I fit the model/strategy parameters using the data from the past, and then I use this fitted model to trade for certain period of time (let's say a month). After this period, I add data from the most recent past period and refit the model. For visualisation of such pipeline, see picture below:
Is such approach considered IS or OOS? (My intuition is that it is OOS, however my intuition is also that this is the most natural way to perform a backtest, which seems not to be the case based on De Prado's paper).