strategy for backtesting

Question

Short version: How do I evaluate models that predict stock performance?

Back story: I'm working for a stock analyst, writing an application to sort through stocks and find candidates that fit the criteria he is looking for. The point is to bring stocks to his attention so he can do fundamental analysis.

For example, he has a mechanism for calculating the expected price range, then we look at where it is currently in that range. There are various factors that might put pressure on the stock to move up or down. Each of his models gives the various factors a different weighting.

We want to backtest the models against the last ten years of data, including the recession. For the top and bottom 25 stocks for a given model projection, I need to score how the stocks did over the next period. Repeat. I cumulatively score the models and see which give the best predictions.

My question is how best to score the stock performance.
Do I take into account that they should sell if the stock price goes down e.g. > 10% of the price range or that the stop-loss should ratchet up? But those have more to do with the buying/selling scheme, nothing to do with the projection model.
In this case, I just want to evaluate model against model.

Do I compare price rise to change of the entire market? Or just within industry? Or compare against the price range for that stock?

There are lots of sites that allow you to do backtesting, but that doesn't answer my question about how to score the models.

The article on evaluating Bollinger Bands comes the closest of anything I've seen to touching on the topic but still manages to avoid it.

Answer 1

Split the strategy into predictors and outcomes, then evaluate how well the predictor predicts the outcome. Don't backtest a trading strategy, evaluate an estimator/predictor using typical machine learning/stats.

If the position within a range is supposed to predict whether a stock will go up or down after 10 days then just test that; measure the rank correlations between range decile and 10 day future return decile or whatever.

Don't try and mix an entire trading strategy and backtest repeatedly; you won't know what you are measuring and you'll overfit to the historical data.