Index reconstruction

This is quite a rookie question. I've searched for solutions but have no luck finding the exact same question...

I am going to do some research on historical prices/returns of the underlying stocks of an index, e.g. S&P 500/Russell 3000. However, the indices are always reconstructed at a point of time, for example ticker A was in the index but then kicked out. How should I structure/modify the dataset? Does it make sense to include all companies that have appeared in the index and just assign returns to be 0s when they are not? What is the industry convention on this?

Edits: Please note I'm not trying to replicate the index return, but to test strategies using the constituents' returns.

Thanks!

Perhaps you just need to treat the index like a simulated portfolio.

Simplistically, if a constituent enters the index, you add the it to your portfolio. If it exits the index, you sell it.

Prior (or post) a stock being in the index is irrelevant. If you did incorporate such periods, this would introduce anomalies such as survivorship bias and look-ahead bias.

If you're not holding it, there is no exposure and therefore no return.

Indexes are somewhat more complex than this though - they are weighted on various other factors including market cap, liquidity, free-float, cap limits etc. The index "re-weights" itself every day on the close.

• Thanks for the comments. The index won't be a problem since I would only focus on constituents. But since I have other metrics to determine buy/sell points of the stocks I don't know if your method works... I am not trying to replicate returns of the indices or anything like that but to test some strategies by trading the underlying Mar 22 '18 at 15:48
• There are many ways to do it. If you already store daily data for each ticker (ex. price) then you can add another field to your data, i.e. a true or false flag that indicates whether this ticker on this date was a member of the S&P500 or not. This is how many commercial stock databases are organized. When you program your backtest, one of the criteria for buying is that the S&P500 flag is true. Mar 22 '18 at 21:24

The replicating methodology depends on the index you are trying to replicate. In most instances, investors try to replicate some variation of a float adjusted capitalization weighted (i.e., float weighted) total return index (i.e., most S&P indices).

In this case, you can very closely replicate the total return of the index if you know for all times a) what the constituents are; b) what the float adjusted capitalizations are; c) the ADJUSTED price changes of the constituents. If you are using adjusted prices, you do not have to keep a tally of dividends and splits since this is reflected. Also, you do not need to keep track of former or future constituents since the only thing which matters to the return is the float weighted price change.

To illustrate, S&P indices typically define the change in the index by a Laspeyres index:

$\frac{I + \Delta I}{I} = \frac{\sum_i P_{i,1}*Q_{i,0}}{\sum P_{i,0}*Q_{i,0}} \,; \forall i \in I$

where: $I$ is the index level; $P_i$ is the price of asset $i$; and, $Q_i$ is the float adjusted share count of asset $i$.

Please reference this following S&P document for a more robust definition: http://us.spindices.com/documents/methodologies/methodology-index-math.pdf

Total Returns Indices are further defined as follows:

$\frac{I_{TR,t}}{I_{TR,t-1}} = \frac{I_{t-1} + \Delta I_t + \sum_{i,t} (D_{i,t}*Q_{i,t})}{I_{TR,t-1}}$

where: $I_{TR}$ is the total return index level; and, $D_{i,t}$ is the dividend for asset $i$ on dividend ex-date $t$.

So, to answer your question, the ability to replicate the index depends on having the right data. If, for example, you cannot infer float adjusted weights, you will not be able to accurately replicate the S&P 500 or similar index.

• Thanks Daivd, but I'm not trying to replicate the index. Sorry for not clarifying my question - pls see the edits. Mar 22 '18 at 17:01
• In that case, I would say that best practice is to build your database as though it were “point in time”, thus reflecting only the information that was known up to a certain date. Rather than making the returns zero (which can lead to all sorts of hazards), it is better to have a Boolean operator which flags securities in your universe as constituents of certain indices at a given point in time. Mar 23 '18 at 0:19