# Tag Info

## Hot answers tagged backtesting

30

Here are a few risks when using historical data: Data fidelity: Is your data an accurate reflection of history? For stocks, should you use actual closing prices or adjusted prices? For futures, how should you construct a realistic, continuous contract? Simulation realism: Are you making realistic assumptions about trade execution? Are you naively assuming, ...

21

Consider the standard error, and in particular the distance between the upper and lower limits: $$\Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha$$ Using the formula for standard error, we can solve for sample size: n = \left(\frac{2 \cdot s \cdot ...

18

Strictly speaking, data snooping is not the same as in-sample vs out-of-sample model selection and testing, but has to deal with sequential or multiple tests of hypothesis based on the same data set. To quote Halbert White: Data snooping occurs when a given set of data is used more than once for purposes of inference or model selection. When such ...

16

Building an effective backtest is not significantly different than building any other kind of predictive model. The goal is to have similar behavior out of sample as you have in sample. As such, there are methodologies developed in statistics and machine learning that can be useful: Understand the bias/variance tradeoff. This is covered in many places. ...

14

I unfortunately can't point you to a great book on the exact subject that you're describing. The closest thing for beginners is "Quantitative Trading". It's a reasonable introduction, but I really wouldn't recommend it as a primary source. The author is at best incomplete (if not misleading) on a number of issues. My favorite book at the moment is ...

11

For "maximum pessimism" you should calculate thus: for longs - enter at the bar high and exit at the bar low on bar following signal bar for shorts - enter at the bar low and exit at the bar high on bar following signal bar I had previously heard this approach to back testing called the "torture test."

11

Accuracy The trader must make sure the data is not only right, but that the timestamps are useable. That's why a good data warehouse will be bitemporal or point-in-time. Thus, we know not only when the item was announced, but when we received it and could act on it. Gaps An aggressive safety check on incoming data might inadvertently exclude correct data. ...

11

A few pointers: When I looked into this a few years ago, a good solution at the time was LIM's XMIM, which also has an S-Plus/Matlab interface. Whit Armstrong also provided an R package for this, although I don't know how complete it is. This provides both the data and the software for analysis. On the very high end (and expensive) side of the spectrum, ...

11

You're not really asking how to backtest a strategy. You already have run a backtest to generate simulated trades. What you're asking for is a way to assess the performance of those simulated trades. You can do this with the R package blotter. You'll need to setup your account and portfolio, then loop over each row in your CSV and call addTxn. For ...

10

I find this one very helpful: Re-Examining the Hidden Costs of the Stop-Loss by Wilson Ma, Guy Morita, Kira Detko Abstract: In this paper, we present general implications of the impact of stop-losses to future returns. The use of stop-losses change return distributions, but not in the way that one would typically expect. We find that while ...

9

Knowledge leads to profit BUT NOT the reverse The scientific method is your friend here. Wandering away from reality into fantasy is so tempting but will lead to failure. Step away from the P&L! Question -> observe -> theory -> predict -> measure -> record/publish/peer review -> repeat Think long and hard about what you believe are the facts/processes ...

9

I did some digging and found the following papers - most of them offering quite a distinct perspective compared to classical option pricing theory! Stock Options as Lotteries by Brian H. Boyer et al. (2011) The Efficiency of the Buy-Write Strategy: Evidence from Australia by Tafadzwa Mugwagwa et al. (2010) The following is my favorite: You could do some ...

9

I'll not say how most people do it, but rather how I think most people should do it. You should compare the actual strategy with a number of goes of randomly trading through the time period using the same constraints as the strategy. Basically this is a way of not mixing species of fruit and seeing what the distribution of luck is for the particular fruit ...

8

Some LinkedIn groups are particularily adequate to post these questions. Your question has already been asked in "Automated Trading Strategies" at this URL: Seeking input on QuantFactory, Deltix and 4thStory: Professional end-to-end Automated Trading Solutions. Feel free to let us know the state of your research.

8

I think the biggest risk is trusting your model too much. I would summarize modelling like that: Model for the best but risk-manage for the worst! As an example for modelling a portfolio approach with derivatives that could e.g. mean: use black scholes for option pricing (model) but manage your risk by assuming a power law distribution and vary your alpha ...

8

I have seen Hansen's SPA ('Superior Predictive Ability') test and stepwise variants used for this purpose. Hansen's test is a Studentized version of White's Reality Check. The stepwise variants allow one to accept or reject the null of no predictive ability on a subset of some tested strategies while maintaining a familywise error rate. In his book, ...

8

This is an evergreen. I've been discussing this with many people - without any clear-cut conclusion. The answer and the preferred solution depend on your trading style (e.g. frequency), your skills, the size of the team, and many other factors. For simplicity, I call "Research" the Matlab/R/etc. environments, whereas "Live" refers to the re-programmed ...

8

We cannot give you a relative bid-ask spread that would make sense. The reason for that is that it really depends on several parameters: The type of financial asset you invest in (futures, funds, index, options, ...) The period during which you're trading (I think the liquidity in markets hasn't been the same over time). If you trade intraday, it depends ...

7

Slippage is multi-facetted, however, I think the main element to slippage is going to depend on the sophistication of your execution approach. Also, in your case there are 2 types of slippage: execution slippage (i.e. cost above mid to get your fill) tracking slippage (how much price difference between actual close and your fill price) Execution ...

7

There is a huge difference between R (and Matlab, SAS, or other statistical languages) and relatively low-level languages such as C/C++/C#/Java in exactly this regard. The latter category is used more often for stable end-products, where speed and performance can be crucial, whereas the former category is used more often for model testing and prototyping. ...

6

This blog post points to a presentation about backtesting and data snooping: http://www.portfolioprobe.com/2010/11/05/backtesting-almost-wordless/ I think the only non-datasnooping method there is is to trade live. But the problem of data snooping can be reduced by seeing how significant the backtest result is compared to what would have happened if the ...

6

Ensemble methods, or ensemble learning are a class of statistical methods that, loosely speaking, operate on many rather than a single instance of the data. Think bootstrapping, but then combine the estimates for an aggregate. The Wikipedia link has more. Combining two forecasters is something else that is sometimes called pooling forecasts or, more ...

6

Here's a link to daily weather data. It looks like it goes as far back as the 1940s. There's a link to a CSV file at the bottom of the page. It will only give you one year's worth of data at a time, so you'll have to manually download several files. http://www.wunderground.com/history/airport/KNYC/2011/3/13/CustomHistory.html

6

Glad people are reading. Simple with more history in terms of time and indexes is better in my book. I have spent 13 years reading over 200 research papers, incorporating complicated and advanced techniques, and studying very reputable buy side research with no improvement in results. Readers are on their own to extend to lots of markets including Nikkei ...

6

The only benefits I can imagine from re-coding in C++ would be speed. But speed doesn't seem to be a concern for your time horizon (especially if you write efficient Matlab code). Some may argue that C++ is more stable, but Matlab is plenty stable for live trading. However, the downsides of using two languages are even more significant than just ease of ...

6

Interestingly enough there is no scientific theory that suggests what fraction of the data should be assigned to training and testing and results can be very sensitive to these choices. From Quantitative Trading by Ernest Chan (p. 53-54): Out-of-Sample Testing Divide your historical data into two parts. Save the second (more recent) part of the data ...

6

Yes, there is in fact a whole literature on this subject coming from the field of non-linear dynamics-- it is known as the method of surrogates. The idea is essentially to come up with a "scrambled" version of your original data set that preserves many of the basic statistical properties, though perhaps not the serial dependence structure which might be ...

6

http://www.portfolioprobe.com/2010/11/05/backtesting-almost-wordless/ shows an example of how the results from a backtest can be deceiving. This would be true with either returns or value. The main issue is that the portfolio you start with can have an impact on what "good" means.

5

There are many details you need to take into account in order to do a proper backtest. Besides the correction regarding bar entry/exit prices mentioned earlier, you will also need to correct for the bid/ask spread, which may be wider than the high/low for some stocks at some times. Some other details are: Splits Dividends Ticker changes Warrants/rights ...

5

I recommend to use MATLAB / Excel for simplicity - depends which one do you already know. Write down the SDE for geometic brownian motion (to simulate stock price over time) on paper, as quant_dev mentioned. Discretize it using i.e. forward Euler discretization (see Wikipedia), code up a MC simulation to simulate it for the time period you want to price ...

Only top voted, non community-wiki answers of a minimum length are eligible