1. Lets say I for work with data from 2000-now in one sample (in-sample), and lets say that my out-of -sample will be from 2000 to 1950. I will then get some type of out-of-sample result.

  2. If i then run a regression with in-sample data being from 1950, and up til 2000 (meaning that my-out of-sample will be from 2000-now), will i then get the same results from out-of-sample in point 1, as i will get in-sample in point 2?


No, you should not expect to get the same results. It would be a spooky coincidence if they actually coincided. Sufficiently spooky as to warrant an audit if they did!

Rather the question you can ask is the likelihood the distributions of the two sub-samples are drawn from the same or a different population. Pooling the means and and the variances of both, ie looking at 1950-now, then what is the probability that both 1950-2000 and 2000-now come from the same wider population with these wider features? That’s a Kolmogorov-Smirnov test.

Else - simpler -if you were simply interested in whether post-2000 looked different from 1950-2000, then add a dummy variable 0-1 for everything after 2000. And another interactive dummy = 1x your regressor variable after 2000 and 0 before. Or multiple dummies for each regresssor. Regress this for 1950-now with these dummies and without. Then F-test the restriction that pre-2000 looks the same as post-2000.

Take the RSS of the two, and calculate the difference. Divide it by the RSS of the unrestricted, the lower value with with the dummies included. Multiply by (n-k), which is the number of datapoints (n) minus the variables in the model, including the intercept and all your dummies. Divide it by the number of dummies used. Call these “d”,

This is your “F-statistic”. It’s a measure of whether adding all of those dummies was statistically significant or not. Excel or statistical tables will give you a p-value for F(n-k,d). This is the probability you’d randomly get the same results without trying to make post-2090 look different to pre-2000

By (subjective) convention, p below 5% suggests something is “maybe” different. Below 1%, “probably” and below 0.1% “almost certainly” so. The caveat being that a p-value of say 10% still means it’s more likely than not they differ. You just can’t hope to prove it. You cannot be certain they are not the same.

Hope this helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.