How many explanatory variables is too many?

Question

When researching any sort of predictive model, whether using ordinary linear regression or more sophisticated methods such as neural networks or classification and regression trees, there seems to always be a temptation to add in more explanatory variables/factors. The in-sample performance of the model always improves, and sometimes it improves a great deal, even after one has already added quite a few variables already. When is it too much? When is the supposed improvement in in-sample performance very unlikely to carry over into live trading? How can you measure this (beyond simple things like the Akaike and Bayesian Information Criteria, which don't work very well in my experience anyway)? Advice, references, and experiences would all be welcome.

Answer 1

“Make things as simple as possible, but not simpler.” The problem you want to avoid is (near) multicollinearity. The tip-off will be that adding/removing a regressor will significantly change the coefficients on the other regressors. In practice (well, in the research that I read) I rarely see this explicitly tested.

If you think that you have multicollinearity, then it's likely best to either estimate over a subset without multicollinearity or to drop the offending regressors. A model with less explanatory power as measured by $R^2$ is certainly better than a model with incorrect (unstable) explanatory power.

Answer 2

There's no rule to answer this question for you. You need some combination of:

• Judgment: Are the parameters you're including reasonable?

• Sniff test: Is there theory to justify your parameter choices, or are you just hunting for chance associations?

• Hold-outs: You correctly mention that the problem is "in sample performance." The solution is therefore to hold out some data when you start and look at out-of-sample performance. Of course, if you iterate enough times, you can over-fit your holdout sample, too! So save this until the last step, and be honest with yourself.

As always, the key is to be certain of what question you are trying to answer. Then you can muster as much unbiased evidence as possible.

Answer 3

I think you're looking for a metric that quantifies the effectiveness of the added variable(s). Objectively, you want each variable to have correlation to your model estimation output and non-correlation between other variables that may be utilized. If you adjust your $R^2$ metric accordingly (less degrees of freedom per variable) you'll get a reasonable feel for where the limit is for adding more variables (otherwise $R^2$ will just increase and you're back to asking the same question).

Answer 4

Just pick up a decent econometrics book (Gujurati is what I used in school).

If you have multicollinearity, find a dummy variable.

http://en.wikipedia.org/wiki/Coefficient_of_determination#Adjusted_R2 << this should be somewhat helpful.

I have no trading experience, so cum grano salis.

Answer 5

Although not directly related to financial modeling, I've found the following quotation to be very instructive:

"I remember my friend Johnny von Neumann used to say, 'with four parameters I can fit an elephant and with five I can make him wiggle his trunk.'" -- E. Fermi

You may also read this: http://mahalanobis.twoday.net/stories/264091/