17
$\begingroup$

Suppose you are running a portfolio of quantitative strategies and that you develop a new potential strategy to be added to the mix. Assume for simplicity that the new strategy is independent of the existing strategy. The new strategy relies on data which is available going back X years. You proceed by backtesting and optimizing the parameters of the new strategy on an "in-sample" portion of your dataset, while reserving an "out-of-sample" portion for validation. The new strategy's weight in your portfolio will be determined by its out-of-sample performance. Your goal is to maximize your overall Sharpe ratio. What is the ideal ratio of in-sample length to out-of-sample length?

$\endgroup$

4 Answers 4

10
$\begingroup$

Interestingly enough there is no scientific theory that suggests what fraction of the data should be assigned to training and testing and results can be very sensitive to these choices.

From Quantitative Trading by Ernest Chan (p. 53-54):

Out-of-Sample Testing Divide your historical data into two parts. Save the second (more recent) part of the data for out-of-sample testing. When you build the model, optimize the parameters as well as other qualitative decisions on the first portion (called the training set), but test the resulting model on the second portion (called the test set). (The two portions should be roughly equal in size, but if there is insufficient training data, we should at least have one-third as much test data as training data. [...]

For more sophisticated methods see Evidence-based technical analysis by David Aronson p. 321-323.

I would add that once the strategy has been revised to reflect such data it is no longer "out-of-sample" or in other words: If you optimize your strategy "out-of-sample" you will incur data snooping bias via curve fitting nevertheless! Or as Aronson puts it:

the Virginal status of the data reserved for out-of-sample testing has a short life span. It is lost as soon as it is used one time.

$\endgroup$
3
  • $\begingroup$ Thank you for your answer. I am generally a fan of Ernie Chan's book, and his blog as well. He gives good guidelines, but at least in this case he does not provide any justification for his recommendation. If no one else comes up with anything, then I think this would be the accepted answer. $\endgroup$ Jul 22, 2011 at 15:08
  • $\begingroup$ @sheegaon: Thank you. I think, as I wrote above, that the problem is that no scientific theory exists that would justify how big the ratio should be. I double-checked: Prof. Aronson confirms this on page 321. It is a little bit like the confidence level 0.05 - there is no real justification for that either but you have to use some value. And for Chan's recommendation 50:50 - this is always the best guess when you have no justification... $\endgroup$
    – vonjd
    Jul 22, 2011 at 16:18
  • 3
    $\begingroup$ Thanks for pointing out the reference to Aronson. It seems like a very thorough and well-written work, even though I would not have looked at it otherwise because the title is rather off-putting to one who prefers to make scientific investment decisions, whereas technical analysis is typically associated with subjective chart patterns. $\endgroup$ Jul 22, 2011 at 18:36
3
$\begingroup$

I think it depends on many factors: the characteristics of the market, the consistency of the market, the number of parameters, the optimization criteria and methodology, the time frame (whether it trades weekly, hourly, by minute, etc), and even the trading strategy.

One approach to answer this question would be to test on many different assets and compare the results of different in sample/out of sample ratios. For example, if you have a system that trades large cap stocks, then maybe you could test different in sample/out of sample ratios on 100 large cap stocks. Then analyze the average and distribution of Sharpe Ratios for each in sample/out of sample ratio.

I would be looking for not only high average Sharpe Ratio but few low outliers.

By the way, due to the many problems of in sample/out of sample testing, I wonder if there are other approaches that have higher likelihood of producing profitable systems.

$\endgroup$
0
$\begingroup$

Another approach is to use k fold or leave one out cross-validation by splitting the time series into a series of "chunks" according to the k value you want to use.

$\endgroup$
0
$\begingroup$

This is a good question. Rather than randomly selecting the ratio, it should be selected based on below two criteria.

  1. Maximum out of sample testing to check the effectiveness of the strategy
  2. Maximum in sample testing so that the machine learning model has sufficient data to learn from and at the same time there shouldn't be overfitting.

The first goal can be achieved by using K fold cross-validation. In these, the full data set is available for out of sample testing. This method splits your dataset into K equal or close-to-equal parts. Each of these parts is called a “fold”. For example, you can divide your dataset into 4 equal parts namely P1, P2, P3, P4.

The first model M1 is trained on P2, P3, and P4 and tested on P1. The second model is trained on P1, P3, and P4 and tested on P2 and so on. In other words, the model i is trained on the union of all subsets except the ith.

The performance of the model i is tested on the ith part.

When this process is completed, you will end up with four accuracy values, one for each model.

To achieve the second goal you need to keep sufficient data for backtesting. This will vary from applications to applications. But generally, you can keep 3/4th or 75% of data for training.

Thank you!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.