43
$\begingroup$

Suppose one has an idea for a short-horizon trading strategy, which we will define as having an average holding period of under 1 week and a required latency between signal calculation and execution of under 1 minute. This category includes much more than just high-frequency market-making strategies. It also includes statistical arbitrage, news-based trading, trading earnings or economics releases, cross-market arbitrage, short-term reversal/momentum, etc. Before even thinking about trading such a strategy, one would obviously want to backtest it on a sufficiently long data sample.

How much data does one need to acquire in order to be confident that the strategy "works" and is not a statistical fluke? I don't mean confident enough to bet the ranch, but confident enough to assign significant additional resources to forward testing or trading a relatively small amount of capital.

Acquiring data (and not just market price data) could be very expensive or impossible for some signals, such as those based on newer economic or financial time-series. As such, this question is important both for deciding what strategies to investigate and how much to expect to invest on data acquisition.

A complete answer should depend on the expected Information Ratio of the strategy, as a low IR strategy would take a much longer sample to distinguish from noise.

$\endgroup$
42
$\begingroup$

Consider the standard error, and in particular the distance between the upper and lower limits:

\begin{equation} \Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha \end{equation}

Using the formula for standard error, we can solve for sample size:

\begin{equation} n = \left(\frac{2 \cdot s \cdot \alpha}{\Delta}\right)^{2} \end{equation}

where $s$ is the measured standard deviation, which you already have from your IR calculation.


High-frequency Example

I was testing a market-making model recently that was expected to return a couple basis points for each trade and I wanted to be confident that my returns were really positive (ie, not a fluke). So, I chose a distance of 3 bps $(\Delta = .0003)$. My sample's measured standard deviation was 45 bps $(s = .0045)$. For a confidence interval of 95% $(\alpha = 1.96)$, my sample size needs to be $n = 3458$ trades. I would have picked a tighter distance if I had been simulating this model, but I was trading live and I couldn't be too choosy with money on the line.


Low-frequency Example

I imagine that for a low-frequency model that was expected to return 1.5% per month, I'd want maybe 1% as the distance $(\Delta = .01)$. If the hoped-for Sharpe ratio were 3, then the standard deviation would be 1.7% $(s = .017)$, which I came-up with by backing-out the monthly returns. So for a confidence interval of 95% $(\alpha = 1.96)$, I'd need 45 months of data.

$\endgroup$
  • $\begingroup$ Discussion moved to chat $\endgroup$ – chrisaycock Sep 14 '11 at 23:05
  • $\begingroup$ Good answer. Can u post the chat transcript here also for completeness. I get a page not found error for the link above. $\endgroup$ – Suminda Sirinath S. Dharmasena Jan 9 '12 at 3:49
  • $\begingroup$ @SumindaSirinathSalpitikorala I get a "This room has been automatically deleted for inactivity" error. There isn't anything for "completeness" anyway; Tal and I had a back-and-forth about examples that ultimately became the answer you see now. Feel free to look at the edit history to see how different my first draft was. $\endgroup$ – chrisaycock Jan 9 '12 at 13:46
  • 1
    $\begingroup$ how exactly did you get s=1.7% from r=1.5% and SR=3? $\endgroup$ – eyaler Jan 27 '12 at 17:38
  • $\begingroup$ @eyaler Solve for $\sigma$ in $(1.5 \times 12) / (\sigma \times \sqrt{12}) = 3$ $\endgroup$ – Chris Taylor Aug 29 '17 at 20:35
4
$\begingroup$

I would also note that you need to watch out for correlations between data points. (EG ,if you have a data point proving this works for oil company x. Another data point for oil company y may not actually count as separate.)

If you are looking at 5 day holding periods, why not just grab all the EOD data that you can as well.EOD data is obviously not tradeable but can be used as a sanity check for long term trading strategy returns when you do not actually have the data.

$\endgroup$
  • $\begingroup$ Hi Michael RB, welcome to quant.SE and thanks for contributing an answer. Do you have any ideas on how the correlation reduces the confidence? As for EOD data, of course it will be used as appropriate, but the question here is how much intraday data I need for a theoretical strategy. $\endgroup$ – Tal Fishman Sep 14 '11 at 0:40
  • $\begingroup$ honestly, it was just an example. for equities, you might want to remove sector returns/ market returns. etc. $\endgroup$ – Michael WS Sep 14 '11 at 1:36
3
$\begingroup$

Surprisingly, no one mentioned the need to ensure that the trading strategy can survive different types of market conditions. Suppose that you use @chrisaycock's formula and came up with 5000 trades. Well, you could make 5000 trades in a few hours or a 1-2 months with the original criteria. Putting aside amount of data to test on, it could just so happen that the time period you selected to test and collect 5000 trades worth of data 'fit' your strategy well. What about 5001th trade or 6000th trade? I would be asking how would you have handled a 'flash crash' or sudden change in product value? A regulator report that a key license required to legally operate the primary business function has been suspended or revoked effective immediately?

A more precise answer in my opinion and experience is to make sure that you sample enough tick data to cover a variety of market conditions. Pay close attention to volatility surrounding sudden events as well as economic calendar announcements (mainly sharp moves in prices). The algo would need to know how to adjust to handle these moves. So you might initially run it over a couple of years of tick data and note these sharp moves and different flavors of volatility. Then on subsequent runs you can focus on the sharp moves first, then replay on the remaining data to further validate the strategy.

$\endgroup$
0
$\begingroup$

To comprehensively validate a short horizon strategy you will need following

  • Data
  • Length: The quantum of the data should be such that it covers different market scenarios. Some common market scenarios are quantitative easing, taper tantrums, dot com bubble, Asian currency crisis and so on. Therefore, the more the better. However, some experts have a different opinion where they say recent data has more importance and testing on past 2 years data should be sufficient. But I prefer a larger length of data to be comfortable.
  • Width: Preferably the strategy needs to be backtested on a wide variety of securities and markets. If your strategy is not specific to a particular security or market then the chances of the strategy working for a longer period of time is higher. And hence the necessity of these steps. Typically, you can run your strategy on stocks with different market cap, volatility profile, ETFs, Currencies, Commodities and so on.

It is very difficult to find a strategy which works all the time but after doing this analysis, you will have clarity on which market conditions and on which instruments it works.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.