5
$\begingroup$

Adding more variables to a model usually increases its accuracy. However, without adequate analysis it could also lead to curve fitting.

Another question (How much data is needed to validate a short-horizon trading strategy?) received answers related to the statistical significance of the standard error of the model. However, I wonder if anyone has results (or analysis) of what should be the ratio of sample data to dimensions used in a model. My intuition has led me to use at least 30 times more sample data points than variables implemented as dimensions but I am not happy with this approach.

I guess that this would depend on the characteristics of the model (it would be different for linear regressions, SVM, non-linear models, etc. and also dependent on the relationships among the variables used) but is there a general framework for estimating this?

Improve this question
$\endgroup$
3
$\begingroup$

The following is a good way to judge the quality of fits for a model.

http://en.wikipedia.org/wiki/Akaike_information_criterion

Improve this answer
$\endgroup$
  • $\begingroup$ Akaike's formula seems to be a good approach to compare several models in a relative way. However, I wonder if it's possible to discriminate a priori in an absolute manner if you are using too many parameters. $\endgroup$ – Gabriel Gómez Rojo Mar 26 '13 at 19:30
  • $\begingroup$ en.wikipedia.org/wiki/Overfitting I do the above. I have a training sample, a validation sample, and an out of sample $\endgroup$ – Andrew Mar 26 '13 at 22:16
3
$\begingroup$

In full generality this is a very difficult question. The closest you will get to a general framework is Vapnik-Chervonenkis theory. You can read about this in Chapter 7.9 of "The elements of statistical learning" by Hastie, Tibshirani and Friedman which can be downloaded from their website .

But be warned that this is a theoretical approach. Often more heuristic approaches will serve you better. Chapter 7 of the book covers those as well.

Improve this answer
$\endgroup$
  • $\begingroup$ The Elements of Statistical learning seems to be a very good resource, thanks! For the moment, I will wait to mark this answer to see if we can attract more people that might share some heuristics or other frameworks. $\endgroup$ – Gabriel Gómez Rojo Apr 3 '13 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.