Backtest overfitting - in-sample vs out-of-sample

Question

Recently, I read a great paper by De Prado et al. on backtest overfitting problem in Quantitative Finance titled Pseudo-Mathematics and Financial Charlatanism: the Effects of Backtest Overfitting on Out-of-sample Perfomance.

In the first chapter, they define in-sample (IS) and out-of-sample performance (OOS) as follows:

With regards to the measured performance of a backtested strategy, we have to distinguish between two very different readings: in-sample (IS) and out-of-sample (OOS). The IS performance is the one simulated over the sample used in the design of the strategy (also known as "learning period" or "training set" in the machine learning literature). The OOS performance is simulated over a sample not used in the design of the strategy (a.k.a. "testing set"). A backtest is realistic when the IS performance is consistent with the OOS performance.

The definitions above are pretty straight-forward, however what confused me is the message in the paper that most people look at IS performance of backtesting when evaluating different strategies. Is that really the case in finance?

For example, most of the time when I did backtest in the past I used the so called rolling-window approach: I fit the model/strategy parameters using the data from the past, and then I use this fitted model to trade for certain period of time (let's say a month). After this period, I add data from the most recent past period and refit the model. For visualisation of such pipeline, see picture below:

Is such approach considered IS or OOS? (My intuition is that it is OOS, however my intuition is also that this is the most natural way to perform a backtest, which seems not to be the case based on De Prado's paper).

Answer 1

It's not out of sample. This is known as the walk-forward backtest and the problem is that you adjust your model based on the PnL curve. You add improvements to reduce drawdowns and increase returns and thus whilst you are scoring and measuring performance that wasn't used in training, you adjust your model based on the scored data. Thus that data forms part of your validation set and is not out-of-sample.

If you do this - you are likely to overfit.

To get a true reflection of out-of-sample performance, you should keep a holdout sample and only score it a handful of times to measure your OOS performance.

This lays the groundwork for why Lopez de Prado says that:

Backtesting is not a research tool - Feature importance is.

I'd like to refer back to some of the previous comments on cross-validation being a gold standard. Your CV performance is created from your validation set, not your OOS set. The model is trained on the training set and scored on the validation set. To see if it would perform out of sample, you need to pass a true OOS set.

---

Before design starts you would split your data into 2 parts, a training set and a test (OOS) set. You would then go on to apply KFold CV on your training data which would now be split it into a train and validation set. In the context of the walk-forward technique, you are training on all the data (or a rolling window) and then scoring on a validation set, usually your next observation, but this is not out-of-sample.

Answer 2

It really depends on how you're planning to use it.

If you're using the new data to 'retrain' your model to different effect (ie, your parameters change a significant amount), that certainly suggests a model misspecification or at minimum a lack of robustness.

Practically, the approach I've generally heard advocated is your top bar (eg, 2016-Q22019 for training/fitting your model; H22019 as a clean dataset on which to test it). This can be something that's updated, but generally there's some reason to do this (eg, potential regime change, uncharacteristic performance) as opposed to simply having more data.

So, in short, it seems like your approach is both.

Answer 3

I would consider this OOS, being the equivalent of cross-validation (the gold standard in machine learning) for time series data.

Answer 4

Yes the procedure you describe is out of sample training yet what de Prado was trying to emphasize in his paper is that even this kind of procedure can lead to overfit. de Prado speaks about tweaking a certain trading strategy (in-sample) and choosing the one with the biggest Sharpe ratio and then running it out-of-sample.Now even though the in sample Sharpe ratio is maximal this does not guarantee that the same will hold out-of-sample. As a matter of fact this can only be satisfied if the number of curves being tweaked on one hand and the number of returns in the sample over which the algo was being run satisfy certain relationship that in turn stems from Extreme Value theory or the Fisher-Tippet-Gnedenko theorem to be precise. It would be interesting to study that theorem to understand more about overfit.