Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I see that institutions still use backtesting by computing P&Ls over historical data and then compute some aggregating ratios to see whether a trading strategy is good or not even though it is not a rigourous approach at all. I mean, how can a trading strategy that happened to perform well in one sample path be guaranteed to perform as well out of sample ?

share|improve this question
1  
Part of the reason is that it's required by laws. – Student T Feb 28 at 23:32
1  
I know that some regulations require VaR to be backtested. But which law has a passage where it says that trading strategies have to be back tested? Of course it is best practice to do so ;) – Richard Feb 29 at 9:17
up vote 3 down vote accepted

How can a trading strategy that happened to perform well in one sample path be guaranteed to perform as well out of sample?

I think you are having it backwards - this is how I do it:

  1. Intuition about some economic, psychological, behavioral, technical etc. phenomenon.
  2. Trying to make my intuition precise in the form of a hypothesis.
  3. Trying to translate my hypothesis into a trading strategy.
  4. Backtesting the trading strategy.
    • In case it does not work: Falsification of my intuition
    • in case it does work: More tests (robustness), and with some refined intuition back to step 1.

Basically this is how the scientific method works when doing research on the stock market. At least this is how it should be, so I somewhat agree with your insinuation that just data mining stock market data to find something is bad science ("Torture the data until they confess" ;-)

So, yes, it is not perfect - but it is the best we have to try to find the signal in the noise (and there is a lot of noise...)

A good starting point to understand more about this approach is this book:
Evidence-based technical analysis by David Aronson

It explains the whole process (including the complete statistical background).

See for a short summary of important points here: CXO Advisory

See for a comprehensive review here: Automated trading system

share|improve this answer

Mostly because of convention and tradition. As Student T mentioned earlier, part of this is that it is common practice. You report to your clients or managers how well something performed in the past; you cannot report to them how well it performed in the future. You may have thought of some useful forward-looking measures, but unfortunately the adoption rate in finance for these things is extremely slow. We still teach CAPM as the forefront in the leading business schools, even though this was introduced in the 1960s. We still credit novelists like Taleb for "discovering" non-Gaussian and black swan behavior in 2000s, even though the authors of the models that he critiques had themselves introduced jump diffusion models in the 1970s.

That said, I think you are making the implicit assumption here that regime shifts prevent you from applying your models out-of-sample:

How can a trading strategy that happened to perform well in one sample path be guaranteed to perform as well out of sample ?

There are persistent phenomena across all time horizons: The low volatility anomaly, the strong dominance of certain factors in explaining returns, volatility clustering, sector correlations, arbitrage between specific symbols, order book properties. It's the presence of these persistent behaviors that motivate people to attempt to extract insight from past data.

share|improve this answer
    
Very good answer. besides persistance what one can focus on is trying to avoid overfitting in the back test. Then chances are higher that the profits will persist. – Richard Feb 29 at 9:19
2  
"You may have thought of some useful forward-looking measures, but unfortunately the adoption rate in finance for these things is extremely slow." Can you give some example of these forward looking measures that general finance is slow to adopt? Genuinely curious, it has always been my perception that back/forwardtesting has been the practice that almost all practitioners use to validate their trading strategies. – Kevin Pei Feb 29 at 12:52
    
"Mostly because of convention and tradition." - I think this is wrong, please see my answer. – vonjd Feb 29 at 15:20

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.