Prompted in part by this question on data snooping, I would be interested to know:

What are the key risks that should be considered when developing a quantitative strategy based on: (a) historical data or (b) simulated data?

  • $\begingroup$ Nice question, but doesn't it even generalize to risks in modeling in general? $\endgroup$ Feb 3, 2011 at 18:21
  • 1
    $\begingroup$ @Dirk Yes, but I'm doing a little SEO for our site. :) $\endgroup$
    – Shane
    Feb 3, 2011 at 18:22

5 Answers 5


Here are a few risks when using historical data:

  • Data fidelity: Is your data an accurate reflection of history? For stocks, should you use actual closing prices or adjusted prices? For futures, how should you construct a realistic, continuous contract?
  • Simulation realism: Are you making realistic assumptions about trade execution? Are you naively assuming, for example, that you can perfectly execute at the day's closing price? Did you remember frictional costs?
  • Sampling variability: Is your historical sample representative of a wide range of market conditions, or did you (happen to) pick a favorable dataset?
  • Curve fitting: Did you fiddle with too many parameters for too long, eventually finding a model that worked great last year but won't make a penny in the future?
  • Optimism: Your actual profits will likely be only a fraction of your simulated profits. Are you assuming otherwise?
  • Model risk: Even if your model back-tests well, what is it's half-life? How long will it be tradable? Very few ideas work forever.

This is not a theoretical list. I've made all these mistakes personally.

  • 1
    $\begingroup$ The half life is in fact a big problem to me. Is there any way to estimate it? (even roughly) $\endgroup$
    – Zarbouzou
    Feb 3, 2011 at 17:24
  • 1
    $\begingroup$ @Zarbouzou Sounds like a good new question to ask. :) $\endgroup$
    – Shane
    Feb 3, 2011 at 17:38
  • $\begingroup$ Nice post Paul, very wise words! I think you should add a link to your article on swap spreads, as a real example showing how you carried out an analysis for a trading idea. I really enjoyed reading it and have learnt a lot, and others will too I am sure. $\endgroup$
    – Robert
    Feb 7, 2012 at 8:32
  • $\begingroup$ data and technological availability is a big one. For example if you do a 10 year backtest, even after correcting for data snooping and overfitting and selection bias many times earlier periods look great, mainly because some of the data you may have used to build your model was not as freely available during that period, as a result the inefficiency was never exploited. Even if the data was freely available some of the algorithms and computational power available to you may today may not have been available at that time to be able to exploit that anomaly in a cost effective way. $\endgroup$ Mar 9, 2017 at 18:17

Knowledge leads to profit BUT NOT the reverse

YOU are the biggest risk to the process. All the hopes, wishes and bias you come with get in the way of making good decisions. The more you want something to be true, the more you have to kick the tires. So many people try out a bunch of random stuff, find a pattern that has a notional profit and just get blinded by dollar signs. That's when you start to skip over important details and you make all the sort of errors as detailed in the other answers here. Even when you are aware of these risks, they only get lip service while you are too busy grasping at the profit. You need to keep your feet on the ground and make reality based decisions. (Note that when you start to loose money fear takes the place of greed and you continue to make bad decisions)

Remember that in the end your own or your clients wealth will rest on the decisions you make. You will have down days, when you see your wealth falling what will you do? Unless, you have rock solid faith in your process you will start to tinker and go into a death spiral. To have that faith you need to have absolute discipline right from the start.

Luckily, there is a well established approach that has been developed of thousands of years to tackle this problem and it has proved very successful. It's called the Scientific method. At this point people throw up their arms and say that's too hard to understand or I don't have time for this. Well if you aren't prepared to work hard and spend the time, you've already lost. Believe me, I've done this for a long time now and I'm convinced this is the most valuable thing you can learn. It doesn't mean you have to have a PhD, wear a white coat and smoke a pipe.

The scientific method really is your friend here. Wandering away from reality into fantasy is so tempting but will lead to failure. Step away from the P&L! Don't look at the notional dollars instead work out what you know and what you can prove.

Question -> observe -> theory -> predict -> measure -> record/publish/peer review -> repeat

Think long and hard about what you believe are the facts/processes that underlie your strategy or idea. Ask questions - What properties can you measure?, what can you predict?, what is the cause? how long does it last?

Observe the world - yes this requires data but it doesn't require P&L! and probably not trading either. Form a theory on how the bits fit together then test this theory - make prediction and measure their success (accuracy etc. NOT P&L).

Objectively assess this new theory. If you are lucky enough to work in a team then this is where peer review and discussion really pay off. If you are on your own, then you face the hard task of dealing with all your behavioral biases and facing up to reality. Good luck with that.

If you make it to the end. Great job, you have learnt something new about the world or at least have an estimate and some associated uncertainty. Most importantly you can have faith that it's right not just wishful thinking.

Now repeat this process until you've built up a useful body of knowledge - a complete picture regarding your idea. Then and only then do you look at trying to best exploit what you know. This will probably lead to more research.

But finally you should have a strategy which you can finally back-test. As little as possible. With the minimum amount of data. If you've done the right job above you will already know what to expect and there shouldn't be many surprises. Resist the urge to tweak, and if you must - only do it with some careful thought and justification.

If its not the money machine that you expected then you made a mistake upstream - go back there and find out the piece you are missing. Start fiddling now and you're off to la la land again.

Do it right then you get to sleep better at night.

Sorry to ramble, but I really do believe that not sticking to the scientific method is the biggest risk to the process.



The trader must make sure the data is not only right, but that the timestamps are useable. That's why a good data warehouse will be bitemporal or point-in-time. Thus, we know not only when the item was announced, but when we received it and could act on it.


An aggressive safety check on incoming data might inadvertently exclude correct data. For example, a tick-capture that compares today's opening price against yesterday's closing price might exclude legitimate bankruptcy notices.


The desk's manager must guard against data mining and other techniques that can cause look-ahead bias. I worked for one hedge fund that required traders to submit their models weeks before production so they could be backtested again without the benefit of hindsight.


I think the biggest risk is trusting your model too much.

I would summarize modelling like that: Model for the best but risk-manage for the worst!

As an example for modelling a portfolio approach with derivatives that could e.g. mean: use black scholes for option pricing (model) but manage your risk by assuming a power law distribution and vary your alpha to see the effect on your portfolio (simulation for risk-management).

I learned that bit in a joint seminar by Wilmott and Taleb - good practical stuff.

  • 1
    $\begingroup$ This is a very good moto I think. Do you know any documentation on this? $\endgroup$
    – Zarbouzou
    Feb 4, 2011 at 14:29
  • $\begingroup$ @Zarbouzou: No, I made this up myself. If you want to cite this just give me a shout ;-) $\endgroup$
    – vonjd
    Feb 4, 2011 at 15:38

I think you can look at books like "The Evaluation and Optimization of Trading Strategies" for some more insight about developing testing strategies.

Also, don't forget that you, the strategy designer, knows the future and could potentially develop biased strategies. It's hard to unlearn something once you have learned.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.