There are two key concerns (which in practice, may be difficult to distinguish):
- Previous research overestimated an effect.
- The effect shrinks over time.
1. Problems with reproducibility and replicability
Previous research may have found an effect, but was the effect really there? There may be problems with:
- Reproducing results using the same data.
- Replicating a result using new data.
Because of errors or problems in analysis, results may not be reproducible even with the same data. For example, failure to account for cross-sectional correlation in financial return data can easily lead to standard-errors that are off by factor of 10.
Because of what Andrew Gelman calls the garden of forking paths, all the choices a researcher must make can lead even a capable, honest researcher to find "statistically significant" results that aren't replicable in new data. In finance, it's not hard to find a trading strategy which through pure luck would have made obscene amounts of money in the past (and there are many people looking)! If I started a VC fund today that will only invest in startups where the co-founders are named Sergei and Larry, you might rightly question how well a back test is estimating expected returns. The numerous degrees of freedom one has when building trading strategies creates what Gelman would call a garden of forking paths.
These are especially prominent problems in psychology but also pervasive throughout science.
2. The effect shrinks over time
Another issue especially pertinent to trading (and much of social science) is that if an effect is based upon people making errors, there's the potential for people to wisen up once the effect becomes widely enough understood.
If small stocks declined in December and advanced in January due to tax-loss selling and subsequent repurchasing, will those effects persist once more traders and investors are aware?
In the finance context, distinguishing between (1) and (2) may be difficult (or practically, may not even matter). To some extent, everyone in macro-finance uses the same dataset: worldwide financial data. To obtain new data, we must all wait for the passage of time.
Gelman, Andrew and Eric Loken, "The garden of forking paths"
Mclean, David R. and Jeffrey Pontiff, 2016, "Does Academic Research Destroy Stock Return Predictability?" Journal of Finance
Peng, Roger D., 2009, "Reproducible research and Biostatistics," Biostatistics