Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

The trading strategies that are going to backtest well are the ones that pick the winners from the past. For example, if a trading strategy simply bought apple stock it would backtest extremely well. The bias is easy to spot in this specific scenario, but for complex trading strategies there could be hidden bias of this type.

I am wondering if it is possible to eliminate this bias by backtesting in a different manner. What if you used the historical data to build a statistical model of the market or part of it. Such a model could include the risk free rate, historic volatilities, and correlations between stocks, etc. You could then run a monte Carlo simulation where you backtest against multiple, statistically generated historical data. You could use this to develop statistics about how the strategy works in a broader sense.

Any thoughts on this idea?

-- Reference : https://www.quantopian.com/posts/backtesting-thoughts

share|improve this question

2 Answers 2

Have you checked White's "reality test" (White H. A reality check for data snooping. // Econometrica. 2000. № 68. С. 1097–1126.)?

Anyway, when you use Monte-Carlo, you always have a variation of "double hypothesis" issue, noted by Fama: first hypothesis is that your model of the market is right, and the second - that trading rule you test (against your market model) actually adds value. Positive answer to the second question makes sense only when the answer to the first is strict 'yes'. And is the first question falsifiable? Perhaps, no.

So, your results might be interesting to publish, but dangerous to trade, in my opinion.

share|improve this answer
    
Interesting, will have a look at this paper, especially the "Monte Carlo Reality check" p-value ....ssc.wisc.edu/~bhansen/718/White2000.pdf ..Thanks –  SMohan Aug 2 at 19:04
    
Marcos de Prado has a great paper on at least detecting this bias: trajectablog.files.wordpress.com/2013/09/… –  experquisite Aug 3 at 19:02

Backtesting, to me, necessarily involves testing against (realised) history of the securities under question. Wikipedia also seems to support this interpretation. http://en.wikipedia.org/wiki/Backtesting

This history of the realised prices, of the securities under question, was generated by a certain pricing "model" or distribution. If you test against a different hypothetical set of prices for the securities then its not backtesting. You are testing against a realisation of prices which might or might not occur in the future.

So my answer to your specific question is that pure backtesting (as opposed to in-sample, out-of-sample) methods might not ever completely guard against data snooping bias.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.