I came up with an approach for measuring data snooping, or overfitting. My question is whether this approach was published and expanded-on already, or is it new?
My approach relies on the observation that if we generate strategies over and over until one does well, then the distribution of PnLs of the postconditioned strategy has some extra kurtosis. This extra kurtosis arises from the sheer fact that we conditioned on the PnL being large. If we don't do any such conditioning (i.e. if our strategy was not created using snooping) then the kurtosis is as it should be, so there is no extra kurtosis. My approach has the advantage that it does not require any knowledge of the particular structure of the strategy space.
Here are the details of my approach:
Suppose our friend Alice gives us a supposedly-great strategy for investing in the stock market. Suppose we backtest this strategy (for simplicity I'll assume we have a perfect backtester) for 252 days, and the PnLs of this strategy are $P_1,\ldots,P_{252}$. We discover that the total PnL, $P = \sum_i P_i$, is very large, so this strategy looks great.
Now we want to figure out if this strategy is really great, so that it will continue being great in the future, or if this strategy is a result of data snooping (or overfitting), meaning Alice tried a bunch of mediocre strategies until she found one whose backtesting results are great.
We make a simplifying assumption, that for any particular strategy, its series of PnLs over the 252 trading days is normally distributed and independent. (So, in particular, for now we assume no kurtosis is present in the raw PnLs.) Now, suppose that Alice did not engage in data snooping. Then by this simplifying assumption, $P_1,\ldots,P_{252}$ should be independent and normally distributed. On the other hand, if Alice did indeed engage in data snooping, then $P_1,\ldots,P_{252}$ are i.i.d normally distributed (say with mean $0$), but then conditioned on the event that $\sum_i P_i \ge M$. (Here, $M$ is some large constant.) In this case, The random variables $P_1,\ldots,P_{252}$ are dependent, and perhaps we can test that. But, even better: they are not normally distributed anymore: they have some measurable kurtosis. This kurtosis is dependent on the value of $M$ and on the number of samples (in our case, $252$) but, presumably, it can be detected by just drawing a histogram of the $252$ samples.
Of course I made some simplifying assumptions that should be removed, such as the PnLs being normally distributed to start with, and our backtesting procedure being perfect. Furthermore, I assumed that Alice is snooping in a very particular prescribed way: that she tosses away a strategy if it doesn't make at least $M$ dollars, and that she gives us the first strategy she finds that makes $M$ dollars. If she tried to fool us by, say, measuring the kurtosis herself and waiting until she finds a strategy with low empirical kurtosis, then our strategy needs to be more sophisticated.
On the other hand, this approach has the advantage that it does not require any knowledge of the particular structure of the strategy space that Alice explored when she tried to find her strategy. So we don't need access to Alice's search space, only to the particular strategy she found. Better yet, we don't even need access to Alice's strategy itself: Alice can keep her strategy secret, and just truthfully report the results of backtesting her strategy.
Has this approach been explored in the literature? Also, does this approach work, or is one of the simplifying assumptions too simplistic, and is this approach bound to fail?
