When researching any sort of predictive model, whether using ordinary linear regression or more sophisticated methods such as neural networks or classification and regression trees, there seems to always be a temptation to add in more explanatory variables/factors. The in-sample performance of the model always improves, and sometimes it improves a great deal, even after one has already added quite a few variables already. When is it too much? When is the supposed improvement in in-sample performance very unlikely to carry over into live trading? How can you measure this (beyond simple things like the Akaike and Bayesian Information Criteria, which don't work very well in my experience anyway)? Advice, references, and experiences would all be welcome.


“Make things as simple as possible, but not simpler.” The problem you want to avoid is (near) multicollinearity. The tip-off will be that adding/removing a regressor will significantly change the coefficients on the other regressors. In practice (well, in the research that I read) I rarely see this explicitly tested.

If you think that you have multicollinearity, then it's likely best to either estimate over a subset without multicollinearity or to drop the offending regressors. A model with less explanatory power as measured by $R^2$ is certainly better than a model with incorrect (unstable) explanatory power.

  • $\begingroup$ This is a good start, but ultimately it's still in-sample. If what you're interested in is prediction, it's hard to beat actually predicting out-of-sample and seeing how good it gets. Since you don't care about the coefficients just the results, avoiding multicollinearity isn't really the answer here. $\endgroup$ – Ari B. Friedman Aug 14 '11 at 9:08
  • $\begingroup$ @gsk3 -- Isn't everything in sample? I don't know tomorrow's data and I would use all (relevant) available data to calibrate my model. If my model is correct, then it should work in sub-samples, unless I think there are multiple regimes, but then it should work in sub-samples of the relevant regime. $\endgroup$ – Richard Herron Aug 15 '11 at 9:38
  • 1
    $\begingroup$ @gsk3 -- And this wouldn't set the upper limit on the number of factors/regressors. I could add humidity downtown and humidity midtown as regressors in my model and "improve" its explanatory power, although these almost certainly have no impact on my model. Because these are collinear, I could get economically and statistically significant coefficients on these factors, even though a change in humidity has no impact on the market. $\endgroup$ – Richard Herron Aug 15 '11 at 9:47
  • $\begingroup$ You want to know how it performs out-of-sample. Therefore you agree in advance that when developing your model you will only use, say 2/3 of your data. That tricks nature into giving you 1/3 of your data as out-of-sample data that you actually can observe. This is a common technique in predictive applications, and it works quite well when you've got a lot of data--as is likely the case here. $\endgroup$ – Ari B. Friedman Aug 15 '11 at 9:50
  • $\begingroup$ The point is that you arrived at humidity down/uptown by trying out a bunch of regressors and chancing on one that fit that dataset. Therefore, the holdout sample is unlikely to show the regressor as being good, and you'll see the problem. If it does work in the holdout sample, maybe you've just found a new regressor that actually does work. It doesn't matter whether it's causal or not--for prediction covariance is good enough as long as it consistently covaries. $\endgroup$ – Ari B. Friedman Aug 15 '11 at 10:57

Although not directly related to financial modeling, I've found the following quotation to be very instructive:

"I remember my friend Johnny von Neumann used to say, 'with four parameters I can fit an elephant and with five I can make him wiggle his trunk.'" -- E. Fermi

You may also read this: http://mahalanobis.twoday.net/stories/264091/

  • $\begingroup$ Funny, but not really an answer. Can you please make this a comment on my question? This is actually a serious question I am facing in my work right now and would appreciate serious answers on this site. $\endgroup$ – Tal Fishman Aug 19 '11 at 20:34

There's no rule to answer this question for you. You need some combination of:

  • Judgment: Are the parameters you're including reasonable?

  • Sniff test: Is there theory to justify your parameter choices, or are you just hunting for chance associations?

  • Hold-outs: You correctly mention that the problem is "in sample performance." The solution is therefore to hold out some data when you start and look at out-of-sample performance. Of course, if you iterate enough times, you can over-fit your holdout sample, too! So save this until the last step, and be honest with yourself.

As always, the key is to be certain of what question you are trying to answer. Then you can muster as much unbiased evidence as possible.


I think you're looking for a metric that quantifies the effectiveness of the added variable(s). Objectively, you want each variable to have correlation to your model estimation output and non-correlation between other variables that may be utilized. If you adjust your $R^2$ metric accordingly (less degrees of freedom per variable) you'll get a reasonable feel for where the limit is for adding more variables (otherwise $R^2$ will just increase and you're back to asking the same question).

  • $\begingroup$ Hi WaveRider, welcome to Quant.SE and thanks for your answer. $\endgroup$ – Tal Fishman Sep 2 '11 at 14:16

Just pick up a decent econometrics book (Gujurati is what I used in school).

If you have multicollinearity, find a dummy variable.

http://en.wikipedia.org/wiki/Coefficient_of_determination#Adjusted_R2 << this should be somewhat helpful.

I have no trading experience, so cum grano salis.

  • 2
    $\begingroup$ I am not the person who downvoted this question, but I am puzzled at your statement. Why should we find a dummy variable when we have multicollinearity in a model? Under what conditions? $\endgroup$ – Tae-Sung Shin Aug 29 '11 at 6:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.