Survival bias when backtesting

Question

I have been doing backtesting, and I am seeking to see if there are any flaws in my program, as it seems to be too good to be true.

Based on stocks with market capitalization of > 10B, go back in times say 20 years and backtest. For each stock, Look at the historical data, stock chart morphological feature, and other features, do a bunch of calculation, assign it a score. On each period, it will pick stocks with good scores.

One of the thing I see is that there can be survival bias. The list of stocks I screen have a market capitalization of > 10B TODAY. So only those who survive today is included. Those who goes out of business were secretly gone without me knowing. Those who survive and become great enough to have 10B capitalization are included. So that's far from ideal.

If it is possible, I would like to get a list of stocks with market capitalization > 10B 20 years ago. But where do I get that list of stock? Is there other ways to avoid that bias?

Answer 1

Trying to determine the historical market cap is difficult (especially with mergers/acquisitions/demergers and multiple share classes with different levels of ownership/voting). Another issue with looking at a fixed market cap level is that it's providing a form of selection bias. The further back you go in time, the less stocks will be included due to the effects of inflation. You could adjust this level back in time using an inflation index, but you might also want to consider market cap in relation to the size of the overall economy.

An alternative strategy here is use stocks in a particular index that suit your market cap parameters. You'd need delisted stocks and index constituent / membership data too.

Many indices have a market cap range built into their methodology:

• S&P 100 (top 100)
• S&P 500 (top 500)
• S&P MidCap 400 (501-900)
• S&P SmallCap 600 (901-1500)
• Russell 1000 (top 1000)
• Russell 2000 (1001-3000)
• Russell 3000 (top 3000)

Disclosure: Norgate Data provides capabilities in this area.

Answer 2

I don't know your budget, and sadly, high quality financial data is not free. There are several good providers of this data. You are looking at spending low to mid 5 figures annually for something like Bloomberg, Reuters, Factset, S&P Global. You can spend a lot less for other providers (Quandl may or may not have what you are looking for), but your mileage will vary.

Answer 3

Others have already suggested that a practical way is to use the composition of a suitable index for your investment universe (to be really safe, the index should have been live at the relevant point in history).

Let me add two remarks. First, the bias you describe is often large. It is studied for US stocks in this paper:

@ARTICLE{Daniel2009,
  author       = {Gilles Daniel and Didier Sornette and Peter W{\"o}hrmann},
  title        = {Look-Ahead Benchmark Bias in Portfolio Performance Evaluation},
  year         = 2009,
  volume       = 36,
  number       = 1,
  journal      = {Journal of Portfolio Management},
  pages        = {121--130}
}

And the authors find that the bias is up to 8% p.a. We looked at this bias for German stocks in Risk-Reward Ratio Optimisation (Revisited), and we found it to be of similar magnitude (about 7% p.a.).

Second, on fixing an absolute size threshold. Better would be to link this threshold to a quantile of market cap. For instance, for US equities Kenneth French publishes percentiles of market cap for NYSE stocks. The following plot shows the evolution of those percentiles.

Your 10bn (shown as the horizontal line) would be around the 75th percentile right now. But if you went back to the 1990s, you wouldn't have too many stocks then. At the start of 1999, for instance, the 75th percentile would have been rather about 3bn or so.

Here would be the R code to reproduce the figure.

library("NMOF")
library("zoo")
bp <- French(dest.dir = "~/Downloads/French",
             dataset = "ME_Breakpoints_CSV.zip")

## make zoo series; scale to millions USD
bp <- zoo(bp[, -c(1, ncol(bp))]/1000000, as.Date(row.names(bp)))

par(mar = c(2,5,1,2), , mgp = c(3.5,0.5,0),
    las = 1, bty = "n", tck = 0.01)
plot(bp,
     plot.type = "single",
     log = "y",
     col = hcl.colors(30, palette = "Grays"),
     ylab = "Market cap in millions USD",
     xaxt = "n",
     yaxt = "n")
mtext(text = colnames(bp),
      side = 4, at = coredata(tail(bp,1)),
      line = -0.7, cex = 0.7)

years <- seq(as.Date("1920-1-1"), as.Date("2020-1-1"), by = "20 year")
bn10 <- 10000000000/1000000
axis(1, at = years, labels = format(years, "%Y"))
axis(2, at = c(axTicks(2), bn10))
abline(h = bn10)

Answer 4

Depending on your strategy, while something to be aware of, the impact of something like this is likely to be minimal. At a 10B market cap, you're well into midcap territory and the companies have already been around a number of years, so it isn't like you're cherry picking small high growth biotech stocks.

Difficult to provide much more absent greater detail as to what you're doing.