2
$\begingroup$

I don't do any financial trading, and this question came up to me as I was browsing through some questions on this SE.

A general sentiment I felt after reading though several questions on this SE, is that algorithmic trading strategies generally won't remain as effective as they were during backtesting, or after some time in deployment.

On some intuitive level I can understand why that might be the case, it just seems odd. But I was wondering if there is some in-depth reasoning on why this is the case.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ the markets will forever be the same in that they will keep changing over time. Traders/investors must adapt to the changes...what works now will not work tomorrow. $\endgroup$
    – Rime
    Aug 29, 2017 at 4:39
  • $\begingroup$ There are several trading strategies that are consistently profitable over the years if incorporate source of data beyond simple time series and action beyond ordinary buy/sell. $\endgroup$
    – columbus
    Aug 30, 2017 at 0:52
  • $\begingroup$ @columbus What? $\endgroup$
    – Ted Taylor of Life
    Sep 2, 2017 at 22:19

1 Answer 1

Reset to default
15
$\begingroup$

There are two key concerns (which in practice, may be difficult to distinguish):

  1. Previous research overestimated an effect.
  2. The effect shrinks over time.

1. Problems with reproducibility and replicability

Previous research may have found an effect, but was the effect really there? There may be problems with:

  • Reproducing results using the same data.
  • Replicating a result using new data.

Because of errors or problems in analysis, results may not be reproducible even with the same data. For example, failure to account for cross-sectional correlation in financial return data can easily lead to standard-errors that are off by factor of 10.

Because of what Andrew Gelman calls the garden of forking paths, all the choices a researcher must make can lead even a capable, honest researcher to find "statistically significant" results that aren't replicable in new data. In finance, it's not hard to find a trading strategy which through pure luck would have made obscene amounts of money in the past (and there are many people looking)! If I started a VC fund today that will only invest in startups where the co-founders are named Sergei and Larry, you might rightly question how well a back test is estimating expected returns. The numerous degrees of freedom one has when building trading strategies creates what Gelman would call a garden of forking paths.

These are especially prominent problems in psychology but also pervasive throughout science.

2. The effect shrinks over time

Another issue especially pertinent to trading (and much of social science) is that if an effect is based upon people making errors, there's the potential for people to wisen up once the effect becomes widely enough understood.

If small stocks declined in December and advanced in January due to tax-loss selling and subsequent repurchasing, will those effects persist once more traders and investors are aware?

In the finance context, distinguishing between (1) and (2) may be difficult (or practically, may not even matter). To some extent, everyone in macro-finance uses the same dataset: worldwide financial data. To obtain new data, we must all wait for the passage of time.

References

Gelman, Andrew and Eric Loken, "The garden of forking paths"

Mclean, David R. and Jeffrey Pontiff, 2016, "Does Academic Research Destroy Stock Return Predictability?" Journal of Finance

Peng, Roger D., 2009, "Reproducible research and Biostatistics," Biostatistics

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.