Selection of optimal backtesting parameters

Suppose I backtest some strategy on in-sample data while varying two parameters, say $$X$$ and $$Y$$. $$X$$ can take the values $$\{3,6,9,12,15,18\}$$ while $$Y$$ can take $$\{10,15,20,25,30\}$$. I want to select appropriate values of $$X$$ and $$Y$$ for testing the strat on out-of-sample data. The tables of sharpe ratios, sortino ratios and max drawdown (dd) are as follows:

    > sharpe
No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
10 0.2923854 0.2485804 0.3116992 0.2356674 0.2711520 0.2535123
15 0.2801226 0.2757317 0.3362495 0.2420944 0.2369459 0.2293062
20 0.2887139 0.2953232 0.2952627 0.2979134 0.2553015 0.2249027
25 0.2736581 0.3268325 0.2971468 0.2896665 0.2401743 0.2240485
30 0.2761537 0.3423867 0.2964909 0.2905532 0.2948999 0.2137761

> sortino
No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
10 0.4080662 0.3380257 0.4144185 0.3087768 0.3521293 0.3231242
15 0.4013694 0.3842653 0.4503256 0.3174395 0.3080369 0.3006281
20 0.4172279 0.4103027 0.3873160 0.3958244 0.3307235 0.2933315
25 0.3925792 0.4787884 0.3940304 0.3848995 0.3095552 0.2892468
30 0.3987750 0.4990707 0.3906656 0.3826982 0.3863327 0.2721931

> dd
No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
10 0.5153225 0.5414108 0.4568199 0.5361848 0.5332630 0.6036963
15 0.4821441 0.4207504 0.3996013 0.5099167 0.5355697 0.5306460
20 0.4246441 0.3970251 0.4178547 0.3985710 0.4945658 0.5100034
25 0.4326678 0.2433439 0.3900689 0.4038422 0.5093099 0.4805518
30 0.3173227 0.2467464 0.3621063 0.3928437 0.3686759 0.4893400


As you can see, the 6-month;30-stock combination gives the best results in general. Someone told me that if, with a small variation in any one of the parameters, the value of the metric changes drastically, I should probably not select those combination of parameters (I guess it could be called an estimation error?). If we move from 6-month;30-stock to 9-month;30-stock, the sortino ratio changes pretty drastically from .499 to .39, and the max drawdown changes from .247 to .362.

So by that logic, I probably need to select the 3-month and 20-stock combination, because the metrics don't change a lot if we vary the no. of stocks or no. of months. My question is: is it okay to just blindly select the combination of parameters that give the best results (6-months,30-stocks) or should I also take the variation in metrics into account (3-months,20-stocks)?

• I have read books and papers that say you should select the parameters in an area of the parameter space where the value function is relatively flat, on the grounds of robustness. But I have never seen a careful analysis of the tradeoffs involved or a proof that this technique is beneficial in reducing out of sample error. I understand the general idea but I have some doubts because in my experience the entire surface changes when you re-estimate it, not just the sharp isolated peaks. The estimation error in these problems tends to be large. – Alex C May 29 '16 at 1:44
• @AlexC : Could you please tell me where I can find those papers or give me their links, in case you remember them? I want to read up on this. – u23 May 30 '16 at 6:11
• any solution on this yet? I have a similar problem and I try to solve it by averaging over both sets of parameter runs. (as in averaging the weights and not the parameters) It works for me as of now but don't know whether that is optimum. – user20964 May 31 '16 at 14:23

Firstly I'm only concentrating on Sharpe, this is the most robust of your metrics. Consider the difference between sharpe and drawdown, sharpe contains a contribution from every return in your result, drawdown is just one observation of whatever happens to be the longest run of negative returns. Of course it's your choice if you want to place more weight on Sortino or drawdown.

Here's the data turned into a heat map, blue highest and red lowest.

This is more of an art than a science so I will attempt to explain what my reasoning would be:

• Roughly as you increase the number of months past 9 things get worse so all other things being equal I would go for something with a lower number of months.
• The lower left triangle of values looks better than the upper right.
• Although there is a local maxima around (15 stocks, 9 months) the other local maxima around (30 stocks, 6 months) looks better. It's also bang in the middle of this better triangle.

Remember you are trying to estimate from noisy observations what the best parameters are. If a small change in a parameter causes a large change in the observed result who knows what sort of "special" circumstance you have wandered into.

In conclusion I'd pick something between 6-9 months and 20-30 stocks. Hopefully you will have some intuition which will help you pick the exact amounts, namely that more stocks or lower periods should be better.

• Thanks a ton! I further performed the backtest for 35 stocks. Even though that added 6 more trials to the backtest, it revealed that for all values of lookback periods, results for 35 stocks are very close to those for 30 stocks. Since there's no substantial variation in the metrics while going from 30 to 35 stocks, I guess it's best to choose the 6-month/30-stock combo. – u23 Jun 2 '16 at 7:41
• Sounds good. Hopefully you also have an out of sample dataset to test these values on. Good luck! – GodLovesATrier Jun 2 '16 at 9:03