# Download historical index data, like SP500, using Python. History going back to at least 1960, if not further back

I want to download historical data for different indices. I am using Python for this. I used the following code.

import pandas_datareader.data as web

start = datetime.datetime(1960, 1, 1)
end = datetime.datetime(2020, 4, 30)

SP500 = web.DataReader(['sp500'], 'fred', start, end)


It gave me data from 2010 onwards.

SP500.head()

sp500
DATE
2010-05-03  1202.26
2010-05-04  1173.60
2010-05-05  1165.90
2010-05-06  1128.15
2010-05-07  1110.88


Is there a way to pull the complete history? Also is it possible to pull data at daily/weekly/monthly/quarterly frequency. It does not have to be datareader, it can be any other function that can accomplish this.

• If this is a repeat question, please point me towards any previous answer. – deb May 2 '20 at 15:15

The solution here is very simple. At any point in time, you will be able to retrieve from the internet stock market data at a daily frequency for free, as long as (1) you do not mind a possible delay of a few hours and (2) having only the past few years of data.

So, all you need to do is to find a dataset today that spans the missing time-frame. Then, you save it somewhere from which you can always retrieve it. Then, you can periodically update this old dataset so that, in practice, you only always need to download the last few days from the internet.

EDIT (following suggestion by noob)

I checked and you can go back to 1927 on Yahoo!Finance: https://finance.yahoo.com/quote/%5EGSPC/history?period1=-1325635200&period2=1588377600&interval=1d&filter=history&frequency=1d

OLD PROPOSAL

I don't know how far back you can get, but I know that Christian Dorion offers his code and his data for an option pricing paper he just published on his website. Buried in his data files are .mat (MATLAB data files) files containing among other things the S&P500 going back at least to 1990. It's not going back to 1960, but it's a good start. Combining this with FRED, averaging 252 trading days per year, that would make up around 7560 data points. Here.

• You can easily find online the S&P going back to 1950 or 1960, just Google. – noob2 May 2 '20 at 19:14
• I don't know why, I just assumed you couldn't get that data. You can go all the way back to 1927 on Yahoo!. – Stéphane May 2 '20 at 19:23
• I was looking for a way to automate that process in python. I agree that I can always save csv file and then load it. I am looking for a way to automate that. And this is not specific to SP500. I want to be able to do it for all possible indices, where if I give the appropriate ticker, startdate, frequency, I get the data back. – deb May 3 '20 at 3:00
• My point, @deb, is that you need only to automate the most recent updates. Some of the data you might want could only be available for just a few years for free. It is fairly possible that you might have to combine many data sources to go back far in time. But it's a none problem because you only need to get the distant past once -- it's only the newest data that you need to append. Get it? – Stéphane May 4 '20 at 2:17
• Thanks. Got it. – deb May 4 '20 at 4:20