"A model estimated with a large no. of observations may allow one to reject null hypothesis of zero coefficients for many explanatory variables.Thus we might choose to select a somewhat lower significance to make rejection of a null hypothesis more difficult." Provide an intuitive explanation for the above statement.
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This is called the "p-value problem in large samples" and s not limited to regression. According to Cohen (1990): "a fact widely understood among statisticians: The null hypothesis, taken literally (and that’s the only way you can take it in formal hypothesis testing), is always false in the real world. If it is false, even to a tiny degree, it must be the case that a large enough sample will produce a significant result and lead to its rejection." This is a fundamental problem with p-values.
Another statistician Chatfield (1995) comments, “The question is not whether differences are ‘significant’ (they nearly always are in large samples), but whether they are interesting. Forget statistical significance, what is the practical significance of the results?” The increased power of large samples means that researchers can detect smaller, subtler, and more complex effects, but relying on p-values alone can lead to claims of support for hypotheses of little or no practical significance.
Looking at practical significance is a better solution than "select a somewhat lower significance to make rejection of a null hypothesis more difficult", although the effect is similar.
I found this article interesting : Mingfeng Lin et al.: "Too Big to Fail: Large Samples and the p-Value Problem".
An analogy might be the comparison of two coins (say 2 pennies): if you have an extremely sensitive balance, able to detect differences of a few atoms, you will find that any two coins you compare are always of different weight. A statistical estimator that uses a large sample is the equivalent of the very precise balance.
