Suppose I backtest some strategy on in-sample data while varying two parameters, say $X$ and $Y$. $X$ can take the values $\{3,6,9,12,15,18\}$ while $Y$ can take $\{10,15,20,25,30\}$. I want to select appropriate values of $X$ and $Y$ for testing the strat on out-of-sample data. The tables of sharpe ratios, sortino ratios and max drawdown (dd) are as follows:

    > sharpe
  No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
            10 0.2923854 0.2485804 0.3116992 0.2356674 0.2711520 0.2535123
            15 0.2801226 0.2757317 0.3362495 0.2420944 0.2369459 0.2293062
            20 0.2887139 0.2953232 0.2952627 0.2979134 0.2553015 0.2249027
            25 0.2736581 0.3268325 0.2971468 0.2896665 0.2401743 0.2240485
            30 0.2761537 0.3423867 0.2964909 0.2905532 0.2948999 0.2137761

> sortino
  No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
            10 0.4080662 0.3380257 0.4144185 0.3087768 0.3521293 0.3231242
            15 0.4013694 0.3842653 0.4503256 0.3174395 0.3080369 0.3006281
            20 0.4172279 0.4103027 0.3873160 0.3958244 0.3307235 0.2933315
            25 0.3925792 0.4787884 0.3940304 0.3848995 0.3095552 0.2892468
            30 0.3987750 0.4990707 0.3906656 0.3826982 0.3863327 0.2721931

> dd
  No_of_stocks  X3.month  X6.month  X9.month X12.month X15.month X18.month
            10 0.5153225 0.5414108 0.4568199 0.5361848 0.5332630 0.6036963
            15 0.4821441 0.4207504 0.3996013 0.5099167 0.5355697 0.5306460
            20 0.4246441 0.3970251 0.4178547 0.3985710 0.4945658 0.5100034
            25 0.4326678 0.2433439 0.3900689 0.4038422 0.5093099 0.4805518
            30 0.3173227 0.2467464 0.3621063 0.3928437 0.3686759 0.4893400

As you can see, the 6-month;30-stock combination gives the best results in general. Someone told me that if, with a small variation in any one of the parameters, the value of the metric changes drastically, I should probably not select those combination of parameters (I guess it could be called an estimation error?). If we move from 6-month;30-stock to 9-month;30-stock, the sortino ratio changes pretty drastically from .499 to .39, and the max drawdown changes from .247 to .362.

So by that logic, I probably need to select the 3-month and 20-stock combination, because the metrics don't change a lot if we vary the no. of stocks or no. of months. My question is: is it okay to just blindly select the combination of parameters that give the best results (6-months,30-stocks) or should I also take the variation in metrics into account (3-months,20-stocks)?

Thanks in advance!

  • 1
    $\begingroup$ I have read books and papers that say you should select the parameters in an area of the parameter space where the value function is relatively flat, on the grounds of robustness. But I have never seen a careful analysis of the tradeoffs involved or a proof that this technique is beneficial in reducing out of sample error. I understand the general idea but I have some doubts because in my experience the entire surface changes when you re-estimate it, not just the sharp isolated peaks. The estimation error in these problems tends to be large. $\endgroup$
    – Alex C
    Commented May 29, 2016 at 1:44
  • $\begingroup$ @AlexC : Could you please tell me where I can find those papers or give me their links, in case you remember them? I want to read up on this. $\endgroup$ Commented May 30, 2016 at 6:11
  • $\begingroup$ any solution on this yet? I have a similar problem and I try to solve it by averaging over both sets of parameter runs. (as in averaging the weights and not the parameters) It works for me as of now but don't know whether that is optimum. $\endgroup$
    – user20964
    Commented May 31, 2016 at 14:23

1 Answer 1


Firstly I'm only concentrating on Sharpe, this is the most robust of your metrics. Consider the difference between sharpe and drawdown, sharpe contains a contribution from every return in your result, drawdown is just one observation of whatever happens to be the longest run of negative returns. Of course it's your choice if you want to place more weight on Sortino or drawdown.

Here's the data turned into a heat map, blue highest and red lowest.

enter image description here

This is more of an art than a science so I will attempt to explain what my reasoning would be:

  • Roughly as you increase the number of months past 9 things get worse so all other things being equal I would go for something with a lower number of months.
  • The lower left triangle of values looks better than the upper right.
  • Although there is a local maxima around (15 stocks, 9 months) the other local maxima around (30 stocks, 6 months) looks better. It's also bang in the middle of this better triangle.

Remember you are trying to estimate from noisy observations what the best parameters are. If a small change in a parameter causes a large change in the observed result who knows what sort of "special" circumstance you have wandered into.

In conclusion I'd pick something between 6-9 months and 20-30 stocks. Hopefully you will have some intuition which will help you pick the exact amounts, namely that more stocks or lower periods should be better.

  • $\begingroup$ Thanks a ton! I further performed the backtest for 35 stocks. Even though that added 6 more trials to the backtest, it revealed that for all values of lookback periods, results for 35 stocks are very close to those for 30 stocks. Since there's no substantial variation in the metrics while going from 30 to 35 stocks, I guess it's best to choose the 6-month/30-stock combo. $\endgroup$ Commented Jun 2, 2016 at 7:41
  • $\begingroup$ Sounds good. Hopefully you also have an out of sample dataset to test these values on. Good luck! $\endgroup$ Commented Jun 2, 2016 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.