Selection of optimal backtesting parameters

Question

Suppose I backtest some strategy on in-sample data while varying two parameters, say $X$ and $Y$. $X$ can take the values $\{3,6,9,12,15,18\}$ while $Y$ can take $\{10,15,20,25,30\}$. I want to select appropriate values of $X$ and $Y$ for testing the strat on out-of-sample data. The tables of sharpe ratios, sortino ratios and max drawdown (dd) are as follows:

    > sharpe
  No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
            10 0.2923854 0.2485804 0.3116992 0.2356674 0.2711520 0.2535123
            15 0.2801226 0.2757317 0.3362495 0.2420944 0.2369459 0.2293062
            20 0.2887139 0.2953232 0.2952627 0.2979134 0.2553015 0.2249027
            25 0.2736581 0.3268325 0.2971468 0.2896665 0.2401743 0.2240485
            30 0.2761537 0.3423867 0.2964909 0.2905532 0.2948999 0.2137761

> sortino
  No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
            10 0.4080662 0.3380257 0.4144185 0.3087768 0.3521293 0.3231242
            15 0.4013694 0.3842653 0.4503256 0.3174395 0.3080369 0.3006281
            20 0.4172279 0.4103027 0.3873160 0.3958244 0.3307235 0.2933315
            25 0.3925792 0.4787884 0.3940304 0.3848995 0.3095552 0.2892468
            30 0.3987750 0.4990707 0.3906656 0.3826982 0.3863327 0.2721931

> dd
  No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
            10 0.5153225 0.5414108 0.4568199 0.5361848 0.5332630 0.6036963
            15 0.4821441 0.4207504 0.3996013 0.5099167 0.5355697 0.5306460
            20 0.4246441 0.3970251 0.4178547 0.3985710 0.4945658 0.5100034
            25 0.4326678 0.2433439 0.3900689 0.4038422 0.5093099 0.4805518
            30 0.3173227 0.2467464 0.3621063 0.3928437 0.3686759 0.4893400

As you can see, the 6-month;30-stock combination gives the best results in general. Someone told me that if, with a small variation in any one of the parameters, the value of the metric changes drastically, I should probably not select those combination of parameters (I guess it could be called an estimation error?). If we move from 6-month;30-stock to 9-month;30-stock, the sortino ratio changes pretty drastically from .499 to .39, and the max drawdown changes from .247 to .362.

So by that logic, I probably need to select the 3-month and 20-stock combination, because the metrics don't change a lot if we vary the no. of stocks or no. of months. My question is: is it okay to just blindly select the combination of parameters that give the best results (6-months,30-stocks) or should I also take the variation in metrics into account (3-months,20-stocks)?

Thanks in advance!

Answer 1

Firstly I'm only concentrating on Sharpe, this is the most robust of your metrics. Consider the difference between sharpe and drawdown, sharpe contains a contribution from every return in your result, drawdown is just one observation of whatever happens to be the longest run of negative returns. Of course it's your choice if you want to place more weight on Sortino or drawdown.

Here's the data turned into a heat map, blue highest and red lowest.

This is more of an art than a science so I will attempt to explain what my reasoning would be:

• Roughly as you increase the number of months past 9 things get worse so all other things being equal I would go for something with a lower number of months.
• The lower left triangle of values looks better than the upper right.
• Although there is a local maxima around (15 stocks, 9 months) the other local maxima around (30 stocks, 6 months) looks better. It's also bang in the middle of this better triangle.

Remember you are trying to estimate from noisy observations what the best parameters are. If a small change in a parameter causes a large change in the observed result who knows what sort of "special" circumstance you have wandered into.

In conclusion I'd pick something between 6-9 months and 20-30 stocks. Hopefully you will have some intuition which will help you pick the exact amounts, namely that more stocks or lower periods should be better.