I have a large dataset (taken from Kaggle: https://www.kaggle.com/borismarjanovic/price-volume-data-for-all-us-stocks-etfs/), and I would like to fill in the missing data.
To do that I can (1) iterate through all the data and get a set of dates that at least one stock was traded on, or (2) get a reference list with dates that trading occurred.
I would like to know from where I can get the data for option (2). Any help?
QuantLib provides calendars for given countries and exchanges, see here.
Dates can then be intersected using something like NumPy's np.intersect1d, for example numpy.intersect1d(cal_dates, numpy.array(db_etfs.loc[:, 'date'])).
@AlexAbrahams's recommended resource is a decent one but here's another (I think better) approach which solves both (1) and (2):
#!/usr/bin/env python
# Call from the `Data` directory
import glob
import pandas as pd
import datetime
# Get all traded dates, with possible repetition
dates = []
for f in glob.glob('*/*.us.txt'):
dates += df['Date'].tolist()
# Remove repetitions
dates = set(dates)
# Optional: Convert to `datetime`
dates = map(lambda s: datetime.datetime.strptime(s, '%Y-%m-%d').date(),
dates)
# Optional: Sort
dates = sorted(dates)
Two reasons to consider doing it this way instead:
You don't need an external, and very large, dependency on QuantLib.
In production, there may be practical, structural or systematic reasons why you can't trade on the days that are in Quantlib but not in your data. For example, the network that your servers are on may be down, causing you to lose data on those dates. Sometimes this loss is connected with events of significant volatility, e.g. a circuit breaker tripping on the exchange causing a glitch in your own software. It doesn't make sense to impute data on the trading schedule "as though" you would've been able to trade on those dates because there's a very repeatable reason why you wouldn't have been able to. Your own data captures these nuances the best as opposed to a third party library.
