What are the key risks to the quantitative strategy development process?

Question

Prompted in part by this question on data snooping, I would be interested to know:

What are the key risks that should be considered when developing a quantitative strategy based on: (a) historical data or (b) simulated data?

Answer 1

Here are a few risks when using historical data:

• Data fidelity: Is your data an accurate reflection of history? For stocks, should you use actual closing prices or adjusted prices? For futures, how should you construct a realistic, continuous contract?
• Simulation realism: Are you making realistic assumptions about trade execution? Are you naively assuming, for example, that you can perfectly execute at the day's closing price? Did you remember frictional costs?
• Sampling variability: Is your historical sample representative of a wide range of market conditions, or did you (happen to) pick a favorable dataset?
• Curve fitting: Did you fiddle with too many parameters for too long, eventually finding a model that worked great last year but won't make a penny in the future?
• Optimism: Your actual profits will likely be only a fraction of your simulated profits. Are you assuming otherwise?
• Model risk: Even if your model back-tests well, what is it's half-life? How long will it be tradable? Very few ideas work forever.

This is not a theoretical list. I've made all these mistakes personally.

Answer 2

Knowledge leads to profit BUT NOT the reverse

YOU are the biggest risk to the process. All the hopes, wishes and bias you come with get in the way of making good decisions. The more you want something to be true, the more you have to kick the tires. So many people try out a bunch of random stuff, find a pattern that has a notional profit and just get blinded by dollar signs. That's when you start to skip over important details and you make all the sort of errors as detailed in the other answers here. Even when you are aware of these risks, they only get lip service while you are too busy grasping at the profit. You need to keep your feet on the ground and make reality based decisions. (Note that when you start to loose money fear takes the place of greed and you continue to make bad decisions)

Remember that in the end your own or your clients wealth will rest on the decisions you make. You will have down days, when you see your wealth falling what will you do? Unless, you have rock solid faith in your process you will start to tinker and go into a death spiral. To have that faith you need to have absolute discipline right from the start.

Luckily, there is a well established approach that has been developed of thousands of years to tackle this problem and it has proved very successful. It's called the Scientific method. At this point people throw up their arms and say that's too hard to understand or I don't have time for this. Well if you aren't prepared to work hard and spend the time, you've already lost. Believe me, I've done this for a long time now and I'm convinced this is the most valuable thing you can learn. It doesn't mean you have to have a PhD, wear a white coat and smoke a pipe.

The scientific method really is your friend here. Wandering away from reality into fantasy is so tempting but will lead to failure. Step away from the P&L! Don't look at the notional dollars instead work out what you know and what you can prove.

Question -> observe -> theory -> predict -> measure -> record/publish/peer review -> repeat

Think long and hard about what you believe are the facts/processes that underlie your strategy or idea. Ask questions - What properties can you measure?, what can you predict?, what is the cause? how long does it last?

Observe the world - yes this requires data but it doesn't require P&L! and probably not trading either. Form a theory on how the bits fit together then test this theory - make prediction and measure their success (accuracy etc. NOT P&L).

Objectively assess this new theory. If you are lucky enough to work in a team then this is where peer review and discussion really pay off. If you are on your own, then you face the hard task of dealing with all your behavioral biases and facing up to reality. Good luck with that.

If you make it to the end. Great job, you have learnt something new about the world or at least have an estimate and some associated uncertainty. Most importantly you can have faith that it's right not just wishful thinking.

Now repeat this process until you've built up a useful body of knowledge - a complete picture regarding your idea. Then and only then do you look at trying to best exploit what you know. This will probably lead to more research.

But finally you should have a strategy which you can finally back-test. As little as possible. With the minimum amount of data. If you've done the right job above you will already know what to expect and there shouldn't be many surprises. Resist the urge to tweak, and if you must - only do it with some careful thought and justification.

If its not the money machine that you expected then you made a mistake upstream - go back there and find out the piece you are missing. Start fiddling now and you're off to la la land again.

Do it right then you get to sleep better at night.

Sorry to ramble, but I really do believe that not sticking to the scientific method is the biggest risk to the process.

Answer 3

Accuracy

The trader must make sure the data is not only right, but that the timestamps are useable. That's why a good data warehouse will be bitemporal or point-in-time. Thus, we know not only when the item was announced, but when we received it and could act on it.

Gaps

An aggressive safety check on incoming data might inadvertently exclude correct data. For example, a tick-capture that compares today's opening price against yesterday's closing price might exclude legitimate bankruptcy notices.

Retro-fitting

The desk's manager must guard against data mining and other techniques that can cause look-ahead bias. I worked for one hedge fund that required traders to submit their models weeks before production so they could be backtested again without the benefit of hindsight.

Answer 4

I think the biggest risk is trusting your model too much.

I would summarize modelling like that: Model for the best but risk-manage for the worst!

As an example for modelling a portfolio approach with derivatives that could e.g. mean: use black scholes for option pricing (model) but manage your risk by assuming a power law distribution and vary your alpha to see the effect on your portfolio (simulation for risk-management).

I learned that bit in a joint seminar by Wilmott and Taleb - good practical stuff.

Answer 5

I think you can look at books like "The Evaluation and Optimization of Trading Strategies" for some more insight about developing testing strategies.

Also, don't forget that you, the strategy designer, knows the future and could potentially develop biased strategies. It's hard to unlearn something once you have learned.