I have a strategy in development that I am backtesting to optimize for parameters, a total of N combinations. Trying my best not to overfit.
I run the first backtest for the in-sample period and I get, say, n results that make money and (N-n) of the rest that don't. I rank the results by return or any other well-defined criteria, and get list L1.
Then I take the top performing p% of n parameters, run another backtest using those parameters on a different out-sample period. I get another similar ranked list, L2.
It's not surprising that L1 and L2 are slightly, and not dramatically, different (L2 is shorter but similar order). What's your next step? Do I take the top m% of L2 and use that for final third backtest on some more data and get L3? Then average the performance rankings of parameters of L2 and L3 use those as a final output? How would you pick what parameters to go with? I am constrained to run only 10 or so parameter sets at best, but I have hundreds that are doing well but are very, very close to each other in performance because some of them cluster around.