108
votes
Accepted
Building Financial Data Time Series Database from scratch
I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as ...
35
votes
Building Financial Data Time Series Database from scratch
All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use ...
25
votes
Accepted
Red Black Trees for Limit Order Book
Why do people suggest using red black trees/balanced binary trees for the levels in a limit order book?
Because people are unoriginal and keep referencing the same blog post.
Why are they ...
17
votes
Accepted
What's the most efficient way to store options and time series data for backtesting?
If the only purpose is to backtest with the data, the primary (in some cases, only) access pattern is to seek to a start time and read all of the data serially through to an end time. Then, there is a ...
16
votes
Building Financial Data Time Series Database from scratch
Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded.
I would like to recommend some ...
15
votes
Building Financial Data Time Series Database from scratch
The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they ...
13
votes
Accepted
Is Yahoo! Finance data good or bad now?
What has changed
The API is gone. The new downloads need 1st to parse the Yahoo Finance page to find a hidden crumb and use it as the key for data retrieval over a 2nd URL.
The adjusted close is ...
12
votes
Accepted
Reliability of CDS indices?
The indices have different quoting conventions.
The way that a CDS index is traded is that you pay a fixed amount per year for protection in case of default (100 bps for IG, 500 bps for HY) and ...
11
votes
Why don't data sources use ISIN instead of symbol?
As a complement to the answer by Enrico, the ISIN identifies a financial instrument in a unique way. Tradition is, of course a reason and many codes, such as the CUSIP code in the US, predate the ISIN....
11
votes
Accepted
Volume or Dollar bars vs. volatility normalized and demeaned financial time series
The dollar bars certainly allow for a partial recovery of normality through a price sampling process subordinated to a volume, tick, dollar clock.
It is well known that returns are assumed to be ...
11
votes
Accepted
How to make a trading universe of liquid futures contracts
Systematically finding most liquid futures instruments
Can we put together a better list than the academic articles?
Yes! The lists in existing publications [1, 2] are great, but fall slightly short ...
10
votes
When looking for arbitrage among a LARGE amount of assets, is there an optimal way?
For example, Thomas H. Cormen, Charles E. Leiserson, Ronald Rivest, Clifford Stein. Introduction to Algorithms, problem 24-3 says:
24-3 Arbitrage
Arbitrage is the use of discrepancies in currency ...
10
votes
Accepted
Efficient way to store orderbook in Python
The best solution is to change how you're doing it completely:
Store the data before it even gets to your program and process it later. e.g. Use your program merely as a daemon to subscribe to the ...
8
votes
Accepted
Practical challenges in storing and managing market data
I've designed large data stores (sitting on about $3-10M of hardware). The challenges that @jharonfe brought up are valid, but I don't feel they're the most interesting challenges in your case (20 ...
8
votes
Where can someone get free (or very cheap) high frequency tick forex data?
Dukascopy offers historical tick data. Through their historical data website you can download what you want, but registration is required, and lots of manual clicking.
However if you are comfortable ...
8
votes
Accepted
Automatically get iShares ETF holdings
No need to scrape the site. That should always be a last resort. The below will import the .csv file you are asking about and save it to a directory of your choice. If you don't want to specify a ...
7
votes
Is "eoddata" a good data source?
I saw a lot of "DON'T USE THEM" messages so I think the OP might benefit from a little more comprehensive answer that puts things in perspective. Firstly, understand that no data source is perfect. I ...
7
votes
Accepted
How to deal with missing value in a time series stock market data?
Some approaches
Use only common points - Exclude all holidays in any index.
Reduced sample size
Loss of information
No 'made up' data (consistency)
Fill forward - use previous day as you ...
7
votes
Accepted
Converting time bars to tick bars or volume bars in python
As mentioned in my comment, tick data is the individual quotes and trades; Yahoo only has daily data. As an analogy, you can always make a high-definition photo more blurry and pixelated, but you can'...
7
votes
Accepted
What are the proper ways to do order book downsampling?
Two chief reasons for subsampling or using a different event space are (i) computational or spatial tractability and (ii) denoising/signal extraction.
Sergei's response seems to focus on the first ...
6
votes
Data exported from Capital IQ, FactSet, Bloomberg, Compustat
Morningstar
Morningstar partnered with Quantopian, and the latter published the structure of Morningstar's equity fundamentals database:
https://www.quantopian.com/help/fundamentals
Quantopian ...
6
votes
Where can I find open swaption implied volatility data?
CME publishes its volatility surface daily on their FTP: CME Vol Surface.
Unfortunately I know of no open APIs that would get you historical data. I'd recommend looking at Barclays Live or Morgan ...
6
votes
Accepted
Does Yahoo/Google no longer support web-scraping of FOREX data?
Yahoo has changed their site structure. The new download URLs look like this:
https://query1.finance.yahoo.com/v7/finance/download/MSFT?period1=1463461200&period2=1494910800&interval=1d&...
6
votes
Which data sources are available for cryptocurrencies?
As mentioned in other answers, Coinmarketcap and Bitcoincharts are two options. The data they provide is also available from Quandl too.
The only issue is, that data has daily frequency and is only ...
6
votes
Accepted
how to treat NA values in Compustat and CRSP
You're going to have to use judgement. There are situations where treating missing values as 0 would be insane. In general, I'd be immensely cautious.
On the other hand, there are situations where it'...
6
votes
Simulating trading strategies
Is there a place online where you can simulate strategies programmatically?
Your best choice is most likely a service such as Quantopian or QuantConnect. Quantopian provides equity and futures data ...
6
votes
Accepted
Using AlphaVantage For Japan/Shanghai/Hong Kong/Shenzhen stock exchange data?
All of the examples just got tested. They all work for me with my API key. Just swap in your API key below, and you should be all set.
For Tokyo:
Sony: https://www.alphavantage.co/query?function=...
6
votes
Px last in Bloomberg
What is px_last is rather complex and will depends on the terminal options installed where you downloaded it. Type DPDF to see what are your settings. You can adjust closing price to account for ...
5
votes
Accepted
OTC Market Data
From what I know OTC http://www.otcmarkets.com/ has their data provided to them by EDGAR, which provides solutions such as API's which can be used to acces things like historical market data. the ...
5
votes
Accepted
Is futures contracts data only reported for business days?
Usually end-of-day prices refer to settlement prices; they are not the last price at which the instrument was trading during that business day. Settlement prices are important because they are the ...
Only top scored, non community-wiki answers of a minimum length are eligible
Related Tags
data × 487market-data × 136
historical-data × 88
equities × 67
programming × 58
time-series × 27
database × 24
finance × 23
bloomberg × 22
tick-data × 22
fundamentals × 20
options × 18
backtesting × 18
index × 16
fixed-income × 15
fx × 15
trading × 12
statistics × 12
quant-trading-strategies × 11
algorithmic-trading × 11
machine-learning × 11
data-mining × 11
futures × 10
high-frequency × 10
etf × 10