Tell me more ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.
up vote 9 down vote favorite
5

Suppose you are running a portfolio of quantitative strategies and that you develop a new potential strategy to be added to the mix. Assume for simplicity that the new strategy is independent of the existing strategy. The new strategy relies on data which is available going back X years. You proceed by backtesting and optimizing the parameters of the new strategy on an "in-sample" portion of your dataset, while reserving an "out-of-sample" portion for validation. The new strategy's weight in your portfolio will be determined by its out-of-sample performance. Your goal is to maximize your overall Sharpe ratio. What is the ideal ratio of in-sample length to out-of-sample length?

share|improve this question

3 Answers

up vote 5 down vote accepted

Interestingly enough there is no scientific theory that suggests what fraction of the data should be assigned to training and testing and results can be very sensitive to these choices.

From Quantitative Trading by Ernest Chan (p. 53-54):

Out-of-Sample Testing Divide your historical data into two parts. Save the second (more recent) part of the data for out-of-sample testing. When you build the model, optimize the parameters as well as other qualitative decisions on the first portion (called the training set), but test the resulting model on the second portion (called the test set). (The two portions should be roughly equal in size, but if there is insufficient training data, we should at least have one-third as much test data as training data. [...]

For more sophisticated methods see Evidence-based technical analysis by David Aronson p. 321-323.

I would add that once the strategy has been revised to reflect such data it is no longer "out-of-sample" or in other words: If you optimize your strategy "out-of-sample" you will incur data snooping bias via curve fitting nevertheless! Or as Aronson puts it:

the Virginal status of the data reserved for out-of-sample testing has a short life span. It is lost as soon as it is used one time.

share|improve this answer
Thank you for your answer. I am generally a fan of Ernie Chan's book, and his blog as well. He gives good guidelines, but at least in this case he does not provide any justification for his recommendation. If no one else comes up with anything, then I think this would be the accepted answer. – Tal Fishman Jul 22 '11 at 15:08
@sheegaon: Thank you. I think, as I wrote above, that the problem is that no scientific theory exists that would justify how big the ratio should be. I double-checked: Prof. Aronson confirms this on page 321. It is a little bit like the confidence level 0.05 - there is no real justification for that either but you have to use some value. And for Chan's recommendation 50:50 - this is always the best guess when you have no justification... – vonjd Jul 22 '11 at 16:18
3  
Thanks for pointing out the reference to Aronson. It seems like a very thorough and well-written work, even though I would not have looked at it otherwise because the title is rather off-putting to one who prefers to make scientific investment decisions, whereas technical analysis is typically associated with subjective chart patterns. – Tal Fishman Jul 22 '11 at 18:36
up vote 0 down vote

Another approach is to use k fold or leave one out cross-validation by splitting the time series into a series of "chunks" according to the k value you want to use.

share|improve this answer
up vote 1 down vote

I think it depends on many factors: the characteristics of the market, the consistency of the market, the number of parameters, the optimization criteria and methodology, the time frame (whether it trades weekly, hourly, by minute, etc), and even the trading strategy.

One approach to answer this question would be to test on many different assets and compare the results of different in sample/out of sample ratios. For example, if you have a system that trades large cap stocks, then maybe you could test different in sample/out of sample ratios on 100 large cap stocks. Then analyze the average and distribution of Sharpe Ratios for each in sample/out of sample ratio.

I would be looking for not only high average Sharpe Ratio but few low outliers.

By the way, due to the many problems of in sample/out of sample testing, I wonder if there are other approaches that have higher likelihood of producing profitable systems.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.