Both for private projects, and for clients, I've been working on code a lot this year to evaluate automated trading strategies. This often ends up turning into the task of how to fairly compare apples and oranges. E.g. to use an FX example, some of the strategies just buy/sell 1 lot, in a single currency. Others are trading in three or four currencies and using different lot sizes, doubling up when losing, etc., etc. Then there are questions about do we consider margin ratios, trading costs, initial capital available. Interest earned when the full capital is not in the strategy? Tax, CGT, stamp duty... don't even go there! Some strategies have different results if we consider average losses of losing trades versus worst loss. (The latter is what decides the margin call.) Which is more important?
But I'm often very disappointed by academic papers that assume no spread, unlimited capital, always getting the advertised price however many lots you want to buy, etc. Some strategies that look great with no spread, fall apart when I apply a pessimistic spread; other strategies are almost unaffected.
So (finally!) my question is could someone point me to papers/books on accepted practice in strategy evaluation and comparison? General advice, discussion and opinion on this topic are also welcome, but the key thing I feel need currently is an "According to Hoyle" reference I can not just use as a guide, but that also clearly explains the trade-offs involved and why most people choose to do it that way.
UPDATE: Thanks for the replies so far, and I'm taking a look at the books suggested here and in other threads. I just wanted to clarify that (for the scope of this question) this is not about designing strategies. I'm being given a set of trades for each of a set of strategies and being asked to say which is best. The trades may have come from algorithms, or from human traders. My preferred approach is detailed simulation: define how much cash at the start, include all costs, and see how much cash at the end. Others seem happy with just counting pip movements. For some strategies this gives a similar result, for others it gives a different result. I want to know how the Big Boys handle this, and why, so I can use that as the basic approach, and then argue intelligently for/against different approaches.