I went through Whites paper of the reality check for multiple strategy testing. To summarize at a simple example:
I have 2 strategies, s1 and s2. s1 gives 2 signals and therefore 2 returns, s2 gives 10 signals and therefore 10 returns.
Here is the Reality Check procedure:
I compare the means of the returns to some benchmark., e.g., mean(returns) = zero. To get p-values of H0, strat returns are zero, I do a Bootstrap of both strat returns, calculate both means, take the max and repeat this prozess a high number of times. I call this array BSvals. From those values I get the p-value of H0(s1) and H0(s2) being the fraction of BSvals values, that is > mean return .
However, In this method I see a flaw. Because of the lower number of signals in s1 the variance of the bootstrapped mean of s1 will be much higher than in s2. Therefore s1 will push pvalues of H0 much higher then if s1 would have signaled more often. Therefore the mean of returns as a performance statistic might not be suitable when comparing strategies with different number of trades/signals. I think of using t statistic as a unbiased performance measure to compensate for different signal numbers, being t = mean(strat returns) * squareroot(number returns) / stddev
How is your thoughts about this? Strat returns being not normal distributed squareroot(number returns) might not be an appropriate scaling factor, though...