5
$\begingroup$

In this famous paper, Bailey and De Prado discard Cross Validation as tool to check for Backtest overfitting, on the ground that it is just an holdout method:

... If we apply the holdout method enough times (say 20 times for a 95% confidence level), false positives are no longer unlikely: They are expected. The more times we apply holdout, the more likely an invalid strategy will pass the test, which will then be published as a single-trial outcome ...

But publishing the results as a single-trial outcome is a misuse of Cross Validation. One should publish the average OOS performance of the K trials. So Bailey and De Prado don't have a point there. Cross Validation does solve the problem of backtest overfitting.

Am I missing something?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ CV is usually abused to become an equivalent of in sample diagnostics. It can help with overfitting when done properly but it doesn’t “solve “ the problem $\endgroup$
    – Aksakal almost surely binary
    Jun 15, 2021 at 19:07
  • $\begingroup$ @Aksakal thank you, could you elaborate a bit more on "helps but does not solve"? By "solve" I meant that CV correctly estimates the OOS performance. What would be a more desirable outcome that correctly estimating the OOS performance? $\endgroup$
    – elemolotiv
    Jun 16, 2021 at 6:19
  • $\begingroup$ CV estimates cross validation performance when done properly. Can it “correctly” estimate it? In social sciences it is a big fat question. Unless you control regressors it is easy to overestimate oos performance with cv $\endgroup$
    – Aksakal almost surely binary
    Jun 16, 2021 at 14:04

1 Answer 1

Reset to default
7
$\begingroup$

If they publish information about all K trials, then you're right. But the author's point is that that's not typical practice. Typical practice is to not disclose that information, and it amounts to p-hacking where the statistical power of the test differs to what's being advertised.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.