Evaluating automated trading strategies: accepted practice

Question

Both for private projects, and for clients, I've been working on code a lot this year to evaluate automated trading strategies. This often ends up turning into the task of how to fairly compare apples and oranges. E.g. to use an FX example, some of the strategies just buy/sell 1 lot, in a single currency. Others are trading in three or four currencies and using different lot sizes, doubling up when losing, etc., etc. Then there are questions about do we consider margin ratios, trading costs, initial capital available. Interest earned when the full capital is not in the strategy? Tax, CGT, stamp duty... don't even go there! Some strategies have different results if we consider average losses of losing trades versus worst loss. (The latter is what decides the margin call.) Which is more important?

But I'm often very disappointed by academic papers that assume no spread, unlimited capital, always getting the advertised price however many lots you want to buy, etc. Some strategies that look great with no spread, fall apart when I apply a pessimistic spread; other strategies are almost unaffected.

So (finally!) my question is could someone point me to papers/books on accepted practice in strategy evaluation and comparison? General advice, discussion and opinion on this topic are also welcome, but the key thing I feel need currently is an "According to Hoyle" reference I can not just use as a guide, but that also clearly explains the trade-offs involved and why most people choose to do it that way.

UPDATE: Thanks for the replies so far, and I'm taking a look at the books suggested here and in other threads. I just wanted to clarify that (for the scope of this question) this is not about designing strategies. I'm being given a set of trades for each of a set of strategies and being asked to say which is best. The trades may have come from algorithms, or from human traders. My preferred approach is detailed simulation: define how much cash at the start, include all costs, and see how much cash at the end. Others seem happy with just counting pip movements. For some strategies this gives a similar result, for others it gives a different result. I want to know how the Big Boys handle this, and why, so I can use that as the basic approach, and then argue intelligently for/against different approaches.

Answer 1

I unfortunately can't point you to a great book on the exact subject that you're describing. The closest thing for beginners is "Quantitative Trading". It's a reasonable introduction, but I really wouldn't recommend it as a primary source. The author is at best incomplete (if not misleading) on a number of issues.

My favorite book at the moment is Expected Returns by Antti Ilamen with a foreward by Cliff Asness of AQR. This really gets into the strategies employed by most quantitative managers, and presents it in a framework that allows you to move forward in your own investigations. That said, while it can serve as a model, it won't directly address methodological issues.

At the end of the day, nothing beats self-understanding. If you want to succeed in quantitative investing, spend the time to understand statistical methods. You can get reasonably far by understanding basic finance (modern portfolio theory, etc.), but a deeper understanding requires knowledge of economics and statistics.

Of course, this site (along with Willmott and NuclearPhynance) can serve as a guide. See "key risks in strategy development" for starters.

Answer 2

I'll not say how most people do it, but rather how I think most people should do it.

You should compare the actual strategy with a number of goes of randomly trading through the time period using the same constraints as the strategy.

Basically this is a way of not mixing species of fruit and seeing what the distribution of luck is for the particular fruit of interest.

More details (including how backtests can easily be misleading) can be found via http://www.portfolioprobe.com/2010/11/05/backtesting-almost-wordless/

Answer 3

You might also try looking at David Aronson's Evidence Based Technical Analysis book and the R ttrTests package. Both outline testing regimes that will enable you to rule out blind luck etc. in system results.

Answer 4

If you are looking for "accepted practice," then in my opinion you must read Grinold and Kahn. That book strongly advocates use of the information ratio, the successor to the closely-related Sharpe ratio. See the wiki pages for lists of alternative ratios and performance evaluation measures, but none are nearly as widespread as these two.

Just one note regarding your question: all performance evaluation should be applied at the portfolio level (i.e. pooling all trades) using mark-to-market (log) returns, probably daily. If a portfolio has no trades, it is holding cash and thus is earning the risk-free rate (which will give a zero excess return for Information/Sharpe ratio). Be careful to always simulate all aspects of your trading without look-ahead bias, including the limitation on the number of trades you can enter (see How to “uncluster” a set of financial data? for an example of how this can be misunderstood).

A number of other questions on this site have already fleshed out various implementation issues, such as:

• Separating the wheat from the chaff: What quant methods separate skillful managers from lucky ones?
• What papers have progressed the field of quantitative finance in recent years (post 2000)?
• Should Sharpe ratio be computed using log returns or relative returns?
• What is the ideal ratio of in-sample length to out-of-sample length?
• What is a reasonable upper bound on the performance of a daily trading strategy?