Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

When researching any sort of predictive model, whether using ordinary linear regression or more sophisticated methods such as neural networks or classification and regression trees, there seems to always be a temptation to add in more explanatory variables/factors. The in-sample performance of the model always improves, and sometimes it improves a great deal, even after one has already added quite a few variables already. When is it too much? When is the supposed improvement in in-sample performance very unlikely to carry over into live trading? How can you measure this (beyond simple things like the Akaike and Bayesian Information Criteria, which don't work very well in my experience anyway)? Advice, references, and experiences would all be welcome.

share|improve this question
add comment

5 Answers

up vote 4 down vote accepted

“Make things as simple as possible, but not simpler.” The problem you want to avoid is (near) multicollinearity. The tip-off will be that adding/removing a regressor will significantly change the coefficients on the other regressors. In practice (well, in the research that I read) I rarely see this explicitly tested.

If you think that you have multicollinearity, then it's likely best to either estimate over a subset without multicollinearity or to drop the offending regressors. A model with less explanatory power as measured by $R^2$ is certainly better than a model with incorrect (unstable) explanatory power.

share|improve this answer
    
This is a good start, but ultimately it's still in-sample. If what you're interested in is prediction, it's hard to beat actually predicting out-of-sample and seeing how good it gets. Since you don't care about the coefficients just the results, avoiding multicollinearity isn't really the answer here. –  Ari B. Friedman Aug 14 '11 at 9:08
    
@gsk3 -- Isn't everything in sample? I don't know tomorrow's data and I would use all (relevant) available data to calibrate my model. If my model is correct, then it should work in sub-samples, unless I think there are multiple regimes, but then it should work in sub-samples of the relevant regime. –  Richard Herron Aug 15 '11 at 9:38
1  
@gsk3 -- And this wouldn't set the upper limit on the number of factors/regressors. I could add humidity downtown and humidity midtown as regressors in my model and "improve" its explanatory power, although these almost certainly have no impact on my model. Because these are collinear, I could get economically and statistically significant coefficients on these factors, even though a change in humidity has no impact on the market. –  Richard Herron Aug 15 '11 at 9:47
    
You want to know how it performs out-of-sample. Therefore you agree in advance that when developing your model you will only use, say 2/3 of your data. That tricks nature into giving you 1/3 of your data as out-of-sample data that you actually can observe. This is a common technique in predictive applications, and it works quite well when you've got a lot of data--as is likely the case here. –  Ari B. Friedman Aug 15 '11 at 9:50
    
The point is that you arrived at humidity down/uptown by trying out a bunch of regressors and chancing on one that fit that dataset. Therefore, the holdout sample is unlikely to show the regressor as being good, and you'll see the problem. If it does work in the holdout sample, maybe you've just found a new regressor that actually does work. It doesn't matter whether it's causal or not--for prediction covariance is good enough as long as it consistently covaries. –  Ari B. Friedman Aug 15 '11 at 10:57
show 5 more comments

There's no rule to answer this question for you. You need some combination of:

  • Judgment: Are the parameters you're including reasonable?

  • Sniff test: Is there theory to justify your parameter choices, or are you just hunting for chance associations?

  • Hold-outs: You correctly mention that the problem is "in sample performance." The solution is therefore to hold out some data when you start and look at out-of-sample performance. Of course, if you iterate enough times, you can over-fit your holdout sample, too! So save this until the last step, and be honest with yourself.

As always, the key is to be certain of what question you are trying to answer. Then you can muster as much unbiased evidence as possible.

share|improve this answer
add comment

I think you're looking for a metric that quantifies the effectiveness of the added variable(s). Objectively, you want each variable to have correlation to your model estimation output and non-correlation between other variables that may be utilized. If you adjust your $R^2$ metric accordingly (less degrees of freedom per variable) you'll get a reasonable feel for where the limit is for adding more variables (otherwise $R^2$ will just increase and you're back to asking the same question).

share|improve this answer
    
Hi WaveRider, welcome to Quant.SE and thanks for your answer. –  Tal Fishman Sep 2 '11 at 14:16
add comment

Just pick up a decent econometrics book (Gujurati is what I used in school).

If you have multicollinearity, find a dummy variable.

http://en.wikipedia.org/wiki/Coefficient_of_determination#Adjusted_R2 << this should be somewhat helpful.

I have no trading experience, so cum grano salis.

share|improve this answer
2  
I am not the person who downvoted this question, but I am puzzled at your statement. Why should we find a dummy variable when we have multicollinearity in a model? Under what conditions? –  Tae-Sung Shin Aug 29 '11 at 6:43
add comment

Although not directly related to financial modeling, I've found the following quotation to be very instructive:

"I remember my friend Johnny von Neumann used to say, 'with four parameters I can fit an elephant and with five I can make him wiggle his trunk.'" -- E. Fermi

You may also read this: http://mahalanobis.twoday.net/stories/264091/

share|improve this answer
    
Funny, but not really an answer. Can you please make this a comment on my question? This is actually a serious question I am facing in my work right now and would appreciate serious answers on this site. –  Tal Fishman Aug 19 '11 at 20:34
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.