1
$\begingroup$

I have a strategy in development that I am backtesting to optimize for parameters, a total of N combinations. Trying my best not to overfit.

I run the first backtest for the in-sample period and I get, say, n results that make money and (N-n) of the rest that don't. I rank the results by return or any other well-defined criteria, and get list L1.

Then I take the top performing p% of n parameters, run another backtest using those parameters on a different out-sample period. I get another similar ranked list, L2.

It's not surprising that L1 and L2 are slightly, and not dramatically, different (L2 is shorter but similar order). What's your next step? Do I take the top m% of L2 and use that for final third backtest on some more data and get L3? Then average the performance rankings of parameters of L2 and L3 use those as a final output? How would you pick what parameters to go with? I am constrained to run only 10 or so parameter sets at best, but I have hundreds that are doing well but are very, very close to each other in performance because some of them cluster around.

$\endgroup$

1 Answer 1

1
$\begingroup$

In my understanding your backtest is introducing a lot of sampling bias. You should use the traditional cross-validation of the training set and once you have calibrated your parameters on this you can try it out of sample. You should rather randomly pick a sample out of the training set, calibrate and validate, but choosing a top% introduces sample bias and is poorly stratified.

$\endgroup$
1
  • $\begingroup$ Thank you for the input. I will try this $\endgroup$
    – CaramelFix
    Jun 20, 2019 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.