I'm backtesting about 1k different strategies / permutations of strategies and I want to identify which if any of the strategies are better than the benchmark.

Based on my readings, I feel like I've narrowed it down to White's Reality Check and the Benjamini-Hochberg-Yekutielie Procedure (with c(M) set to 1). What are the pros/cons of the two approaches? Is one decidedly superior?

Is there a better approach besides these two?



White's approach estimates the dependence structure (simply speaking, the variance-covariance matrix) of the test statistics (such as t-values or p-values) that are used for deciding what strategy is superior. The estimation of the dependence structure is not explicit, but that's what happens during bootstrap or simulation. Typically, the amount of data is not enough to estimate it (think of how much data is required to estimate 1000*1000 variance-covariance matrix reliably) but as far as I know that problem is swept under the carpet in all of the papers of White and his followers.

BH (1995) assumes that the p-values are iid U(0, 1) under the null hypothesis, with a couple of technical caveats that are not going to be of help to you. Then, they also followed the idea of White, i.e. tried working with dependent p-values via resampling in this paper

Yekutieli, D., Benjamini, Y., 1999. Resampling-based false discovery rate controlling multiple test procedures for correlated test statistics. Journal of Statistical planning and inference, 1999

but the limitations of this approach are exactly the same as those of White.

I would suggest using Efron's Fdr/fdr because, in a certain sense, it provides a viable solution when the test statistics are dependent, among other things. You can start with this paper, and the corresponding book can be found on Amazon. The locfdr package is no longer available in R, but I just downloaded and installed it separately.

All of that being said, the ultimate test is seeing how the "best" strategy works out of sample.

  • $\begingroup$ If you were to compare the Benjamini-Hochberg-Yekutieli method and the White's Reality Check method, which one leads to lower type 1 errors? lower type 2 errors? Thanks - I'll look into the Efron paper you recommended. $\endgroup$ – Tarak Oct 18 '14 at 17:48
  • $\begingroup$ It depends. If you look into what Efron did, there is a simple example there that illustrates why. The joint distribution of test statistics can be overdispersed/underdispersed. Then if you apply any method that assumes iid U(0,1) distribution of p-values under the null, the Type I error will be too high / too low compared to the target. $\endgroup$ – James Oct 18 '14 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.