1
$\begingroup$

Short version: How do I evaluate models that predict stock performance?

Back story: I'm working for a stock analyst, writing an application to sort through stocks and find candidates that fit the criteria he is looking for. The point is to bring stocks to his attention so he can do fundamental analysis.

For example, he has a mechanism for calculating the expected price range, then we look at where it is currently in that range. There are various factors that might put pressure on the stock to move up or down. Each of his models gives the various factors a different weighting.

We want to backtest the models against the last ten years of data, including the recession. For the top and bottom 25 stocks for a given model projection, I need to score how the stocks did over the next period. Repeat. I cumulatively score the models and see which give the best predictions.

My question is how best to score the stock performance.
Do I take into account that they should sell if the stock price goes down e.g. > 10% of the price range or that the stop-loss should ratchet up? But those have more to do with the buying/selling scheme, nothing to do with the projection model.
In this case, I just want to evaluate model against model.

Do I compare price rise to change of the entire market? Or just within industry? Or compare against the price range for that stock?

There are lots of sites that allow you to do backtesting, but that doesn't answer my question about how to score the models.

The article on evaluating Bollinger Bands comes the closest of anything I've seen to touching on the topic but still manages to avoid it.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Split the strategy into predictors and outcomes, then evaluate how well the predictor predicts the outcome. Don't backtest a trading strategy, evaluate an estimator/predictor using typical machine learning/stats.

If the position within a range is supposed to predict whether a stock will go up or down after 10 days then just test that; measure the rank correlations between range decile and 10 day future return decile or whatever.

Don't try and mix an entire trading strategy and backtest repeatedly; you won't know what you are measuring and you'll overfit to the historical data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ To more specifically answer your question, the simplest criteria is cost-adjusted risk-adjusted future return, discounted by time. You want to reward models which predict a positive return despite all costs, where the positive return happens immediately and without downside deviation. Ignore stop losses etc, evaluate those last after the portfolio construction etc. $\endgroup$
    – experquisite
    Mar 19, 2017 at 17:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.