When testing your strategy, what you need to pay particular attention to is performance attribution, in other words why did you see the returns you did?

Let me give you a simple example to illustrate what I mean. Suppose I have an algorithm to pick stocks and you have a testing database of stock prices for one year. Suppose also that in that year the market was down 15%. You test my algorithm and find it returned -2%, beating the average by 13%. Is my algorithm a good one?

Now suppose I told you my algorithm was to take your money, charge a 2% management fee off the top and leave all the cash in a non interest bearing account. Does it sound as good now?

The question they is not if a strategy differs from a benchmark index, but rather why?

It could be because:

You selected from a different universe (comparing US equities to the DAX for example).

The risk profile of the stocks you selected differs from the universe (picking only low price stocks out of a universe of all stocks).

Your "universe" differs from the universe from which the index was constructed. For example suppose you select a test dataset consisting of all the NYSE stocks on Jan. 1, 2011 and collect returns for 2011. This would differ from the NYSE index simply because new stocks are listed during the year which were not there on Jan 1, the day you fixed your universe.

Even if these sort of technical factors are fully accounted for, there still remain many great questions for which you would like to have insight. For example if a stock portfolio does well was it:

1. Invested in the right sectors, but underperformed in those sectors?

2. Invested in the under performing sectors but picking the high performing stocks in those sectors?

3. Some permutation of the above?

Bottom line: don't just compare to an index.

When testing your strategy, what you need to pay particular attention to is performance attribution, in other words why did you see the returns you did?

Let me give you a simple example to illustrate what I mean. Suppose I have an algorithm to pick stocks and you have a testing database of stock prices for one year. Suppose also that in that year the market was down 15%. You test my algorithm and find it returned -2%, beating the average by 13%. Is my algorithm a good one?

Now suppose I told you my algorithm was to take your money, charge a 2% management fee off the top and leave all the cash in a non interest bearing account. Does it sound as good now?

The question then is not if a strategy differs from a benchmark index, but rather why?

It could be because:

You selected from a different universe (comparing US equities to the DAX for example).

The risk profile of the stocks you selected differs from the universe (picking only low price stocks out of a universe of all stocks).

Your "universe" differs from the universe from which the index was constructed. For example suppose you select a test dataset consisting of all the NYSE stocks on Jan. 1, 2011 and collect returns for 2011. This would differ from the NYSE index simply because new stocks are listed during the year which were not there on Jan 1, the day you fixed your universe.

Even if these sort of technical factors are fully accounted for, there still remain many great questions for which you would like to have insight. For example if a stock portfolio does well was it:

1. Invested in the right sectors, but underperformed in those sectors?

2. Invested in the under performing sectors but picking the high performing stocks in those sectors?

3. Some permutation of the above?

Bottom line: don't just compare to an index.