The "Shiller P/E regression" refers to the regression of real stock market returns over the next 20 years on the Shiller P/E. When I did this OLS regression myself (based on the data from Prof. Shiller), I got an R-squared of c. 67% and a p value for the regression of essentially zero (corresponding F stat of over 1600).
However, are the aforementioned OLS stats (R-squared, p value) the right ones to think about the goodness of fit and the significance of the regression? I am worried about the issue of overlapping time periods in the sample; i.e. the first item in the sample contains annualized real returns from Jan 1928 to Jan 1948, then the second item contains annualized real returns from Feb 1928 to Feb 1948 - thus the two items in the sample overlap for a whole 19 years and 11 months. Doesn't this somehow lower the "information content" in my sample compared to a sample that were to only contain non-overlapping time periods?
Note that this is an issue mentioned by Prof. Shiller himself in his book Irrational Exuberance (I have the 2nd edition from 2005 in front of me, p. 187):
The relation between price-earnings ratios and subsequent returns appears to be moderately strong, though there are questions about its statistical significance, since there are fewer than twelve nonoverlapping ten-year intervals in the 155 years' worth of data.
In the corresponding footnote, he further writes (p. 261 in my book):
However, the actual academic literature has still not resolved the question of statistical significance. There are unresolved statistical complexities, notably those due to the problem of (near) unit roots in the ratios and the dependency of both independent and dependent variables on price. There are other statistical issues too: a tendency toward rare big outlier observations, issues of the relevance of asymptotic distribution theory in small samples, questions about regime change, and measurement issues for the underlying data, as well as difficulty interpreting complex statistical evidence that has been selectively presented by a researcher who may have a preconceived bias.
He then presents a series of academic papers that address some of these issues and that arrive at differing conclusions. However, most of the papers cited in my edition (2005) are by now a bit dated and hence I am wondering whether a consensus has by now evolved in the academic literature.
(I am aware of this question that deals with overlapping time periods but I was wondering whether any progress has been made specifically on the topic of the "Shiller P/E regression".)