What are the key risks to the quantitative strategy development process?

Question

Prompted in part by this question on data snooping, I would be interested to know:

What are the key risks that should be considered when developing a quantitative strategy based on: (a) historical data or (b) simulated data?

Answer 1

Here are a few risks when using historical data:

• Data fidelity: Is your data an accurate reflection of history? For stocks, should you use actual closing prices or adjusted prices? For futures, how should you construct a realistic, continuous contract?
• Simulation realism: Are you making realistic assumptions about trade execution? Are you naively assuming, for example, that you can perfectly execute at the day's closing price? Did you remember frictional costs?
• Sampling variability: Is your historical sample representative of a wide range of market conditions, or did you (happen to) pick a favorable dataset?
• Curve fitting: Did you fiddle with too many parameters for too long, eventually finding a model that worked great last year but won't make a penny in the future?
• Optimism: Your actual profits will likely be only a fraction of your simulated profits. Are you assuming otherwise?
• Model risk: Even if your model back-tests well, what is it's half-life? How long will it be tradable? Very few ideas work forever.

This is not a theoretical list. I've made all these mistakes personally.

Answer 2

Accuracy

The trader must make sure the data is not only right, but that the timestamps are useable. That's why a good data warehouse will be bitemporal or point-in-time. Thus, we know not only when the item was announced, but when we received it and could act on it.

Gaps

An aggressive safety check on incoming data might inadvertently exclude correct data. For example, a tick-capture that compares today's opening price against yesterday's closing price might exclude legitimate bankruptcy notices.

Retro-fitting

The desk's manager must guard against data mining and other techniques that can cause look-ahead bias. I worked for one hedge fund that required traders to submit their models weeks before production so they could be backtested again without the benefit of hindsight.

Answer 3

Knowledge leads to profit BUT NOT the reverse

The scientific method is your friend here. Wandering away from reality into fantasy is so tempting but will lead to failure. Step away from the P&L!

Question -> observe -> theory -> predict -> measure -> record/publish/peer review -> repeat

Think long and hard about what you believe are the facts/processes that underlie your strategy or idea. Ask questions - What properties can you measure?, what can you predict?, what is the cause? how long does it last?

Observe the world - yes this requires data but it doesn't require P&L! and probably not trading either. Form a theory on how the bits fit together then test this theory - make prediction and measure their success (accuracy etc. NOT P&L).

Objectively assess this new theory. If you are lucky enough to work in a team then this is where peer review and discussion really pay off. If you are on your own, then you face the hard task of dealing with all your behavioral biases and facing up to reality. Good luck with that.

If you make it to the end. Great job, you have learnt something new about the world or at least have an estimate and some associated uncertainty. Most importantly you can have faith that it's right not just wishful thinking.

Now repeat this process until you've built up a useful body of knowledge - a complete picture regarding your idea. Then and only then do you look at trying to best exploit what you know. This will probably lead to more research.

But finally you should have a strategy which you can finally back-test. As little as possible. With the minimum amount of data. If you've done the right job above you will already know what to expect and there shouldn't be many surprises. Resist the urge to tweak, and if you must - only do it with some careful thought and justification.

If its not the money machine that you expected then you made a mistake upstream - go back there and find out the piece you are missing. Start fiddling now and you're off to la la land again.

Do it right then you get to sleep better at night.

Sorry to ramble, but I really do believe that not sticking to the scientific method is the biggest risk to the process.

Answer 4

I think the biggest risk is trusting your model too much.

I would summarize modelling like that: Model for the best but risk-manage for the worst!

As an example for modelling a portfolio approach with derivatives that could e.g. mean: use black scholes for option pricing (model) but manage your risk by assuming a power law distribution and vary your alpha to see the effect on your portfolio (simulation for risk-management).

I learned that bit in a joint seminar by Wilmott and Taleb - good practical stuff.

Answer 5

I think you can look at books like "The Evaluation and Optimization of Trading Strategies" for some more insight about developing testing strategies.

Also, don't forget that you, the strategy designer, knows the future and could potentially develop biased strategies. It's hard to unlearn something once you have learned.