Views on timeframes for backtesting vary considerably. Curious on what timeframe/trade size leads to a statistically significant result. For example, what backtest period is reasonable for a system that trades roundtrip once per day and results in ~250 trades per year? Two years, five years, ten years? Intuitively I struggle with problem because market character/direction also impacts a strategies performance which can span multiple years and so that would suggest the test should span multiple bull and bear markets among other things. So trying to find a reasonable balance that is practical but also statistically significant.
1
-
$\begingroup$ I'm not entirely sure how your backtest procedure will look like, but I think this article may be of interest for you: papers.ssrn.com/sol3/papers.cfm?abstract_id=2308659 They talk about minimal backtest length in a sense to avoid overfitting, which is not entirely what you're asking for. It still might give you some ideas on what period to choose from that point of view. $\endgroup$– Good Guy MikeApr 27, 2014 at 11:45
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
What you are asking is more of an opinion. In fact there is no right or wrong answer perhaps.
The caution with using statistics with real world backtests is that if anything looks good on one parameter, e.g. Sharpe ratio, it will look good on t-stat, p-value and pretty much everything else. The primary reason for backtest is to check if there are periods in the history when the backtests showed poor results in spite of an overall good performance.
The answer perhaps depends on the average holding period of your positions. If your trading strategy's average holding period is 2 months to say 6 months, even if the strategy re-balances (or trades) daily or weekly, it is better to use a long time, preferably 15 years. Definitely include two bear and two bull markets. A long term frame will highlight weaknesses in your models. If you are trading short term strategies that turnover within a 1 week or less, around 7-8 years may suffice. For intra-day or other high frequency strategies, which is probably your case, 3-5 years will do. More than 5 years may actually be incorrect for high frequency strategies as the market microstructure might have changed.
To distinguish backtested strategies that work out of sample versus those that won't, a better thing could be to concentrate on segments of your universe, i.e. will the same strategy work well (albeit with weaker attractiveness) if you take out parts of your universe (e.g. does it work well on large cap stocks OR does it only work well on small cap stocks, etc.).
Lastly, it's very easy to find strategies that worked well for 20 years and just when you start to trade it, it no longer works (in-sample bias). Backtested strategies that tend to work out of sample, in practice, also have an clear edge that you can define. Ask yourself what is the asymmetry in your strategy. Does your strategy makes sense! Example, take a hypothetical rule like "multiply price by 2.3, add 5 and subtract sqrt of volume" - even if this backtested well, there is no reason for it to work.
$\begingroup$
$\endgroup$
I don't think there's a robust statistical approach, without making some arrogant assumptions.
- Because your statistical test window is also the window over which you have done your research, your test size is biased in ways you can not predict. For a statistical test to reliably reflect the chosen $\alpha$, your data not only needs to meet the estimator's assumptions, but you need to have pre-specificed your null and alternative hypotheses ahead of time. This is not the case in trading research: You try $N$ hypotheses, then run your statistical test on the most successful hypothesis. If $N>1$, you suddenly have a test that's heavily biased towards showing statistical significance of your system. I also don't think it's possible to know the extent of this bias of the test statistic since it's not something that increases linearly with $N$ (and there cannot be a concrete definition of $N$).
- You could address this to some small extent by, for example, assuming $\alpha = 0.05$ and do some Monte Carlo with GBMs (correlations within the underlying Wiener processes) or with cointelated processes plugged into your system (note that you will have to plug these GBMs into every system you have ever tried, which means great source code management! I hope you have SVN or Github!).
There is also the trade off between statistical power and overfitting as you extend your test horizon, the former is calculable under distributional assumptions and the latter is not calculable. That is, your power is weaker over shorter horizons, encouraging conservative inference, but overfitting to temporary opportunities (or more likely noise) is more of a problem which encourages a type 1 error.
Of course you need to assume statistical stationarity, and the inefficiency that you have supposedly captured is going to last. Note that this implies that your null and alternative hypotheses that you will be testing are not the hypotheses that you wish to test.
If you don't care about the above, you can use:
How much data is needed to validate a short-horizon trading strategy?
