Hot answers tagged

108 votes
Accepted

Building Financial Data Time Series Database from scratch

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as ...
drobertson's user avatar
  • 1,882
35 votes

Building Financial Data Time Series Database from scratch

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use ...
madilyn's user avatar
  • 5,230
25 votes
Accepted

Red Black Trees for Limit Order Book

Why do people suggest using red black trees/balanced binary trees for the levels in a limit order book? Because people are unoriginal and keep referencing the same blog post. Why are they ...
databento's user avatar
  • 2,478
17 votes
Accepted

What's the most efficient way to store options and time series data for backtesting?

If the only purpose is to backtest with the data, the primary (in some cases, only) access pattern is to seek to a start time and read all of the data serially through to an end time. Then, there is a ...
databento's user avatar
  • 2,478
16 votes

Building Financial Data Time Series Database from scratch

Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded. I would like to recommend some ...
chjortlund's user avatar
15 votes

Building Financial Data Time Series Database from scratch

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they ...
Thomas Browne's user avatar
13 votes
Accepted

Is Yahoo! Finance data good or bad now?

What has changed The API is gone. The new downloads need 1st to parse the Yahoo Finance page to find a hidden crumb and use it as the key for data retrieval over a 2nd URL. The adjusted close is ...
mementum's user avatar
  • 628
12 votes
Accepted

Reliability of CDS indices?

The indices have different quoting conventions. The way that a CDS index is traded is that you pay a fixed amount per year for protection in case of default (100 bps for IG, 500 bps for HY) and ...
Chris Taylor's user avatar
  • 5,901
11 votes

Why don't data sources use ISIN instead of symbol?

As a complement to the answer by Enrico, the ISIN identifies a financial instrument in a unique way. Tradition is, of course a reason and many codes, such as the CUSIP code in the US, predate the ISIN....
fralau's user avatar
  • 321
11 votes
Accepted

Volume or Dollar bars vs. volatility normalized and demeaned financial time series

The dollar bars certainly allow for a partial recovery of normality through a price sampling process subordinated to a volume, tick, dollar clock. It is well known that returns are assumed to be ...
Jacques Joubert's user avatar
11 votes
Accepted

How to make a trading universe of liquid futures contracts

Systematically finding most liquid futures instruments Can we put together a better list than the academic articles? Yes! The lists in existing publications [1, 2] are great, but fall slightly short ...
databento's user avatar
  • 2,478
10 votes

When looking for arbitrage among a LARGE amount of assets, is there an optimal way?

For example, Thomas H. Cormen, Charles E. Leiserson, Ronald Rivest, Clifford Stein. Introduction to Algorithms, problem 24-3 says: 24-3 Arbitrage Arbitrage is the use of discrepancies in currency ...
Dimitri Vulis's user avatar
10 votes
Accepted

Efficient way to store orderbook in Python

The best solution is to change how you're doing it completely: Store the data before it even gets to your program and process it later. e.g. Use your program merely as a daemon to subscribe to the ...
databento's user avatar
  • 2,478
8 votes
Accepted

Practical challenges in storing and managing market data

I've designed large data stores (sitting on about $3-10M of hardware). The challenges that @jharonfe brought up are valid, but I don't feel they're the most interesting challenges in your case (20 ...
madilyn's user avatar
  • 5,230
8 votes

Where can someone get free (or very cheap) high frequency tick forex data?

Dukascopy offers historical tick data. Through their historical data website you can download what you want, but registration is required, and lots of manual clicking. However if you are comfortable ...
Tony's user avatar
  • 181
8 votes
Accepted

Automatically get iShares ETF holdings

No need to scrape the site. That should always be a last resort. The below will import the .csv file you are asking about and save it to a directory of your choice. If you don't want to specify a ...
amdopt's user avatar
  • 4,768
7 votes

Is "eoddata" a good data source?

I saw a lot of "DON'T USE THEM" messages so I think the OP might benefit from a little more comprehensive answer that puts things in perspective. Firstly, understand that no data source is perfect. I ...
user2040657's user avatar
7 votes
Accepted

How to deal with missing value in a time series stock market data?

Some approaches Use only common points - Exclude all holidays in any index. Reduced sample size Loss of information No 'made up' data (consistency) Fill forward - use previous day as you ...
user25064's user avatar
  • 1,077
7 votes
Accepted

Converting time bars to tick bars or volume bars in python

As mentioned in my comment, tick data is the individual quotes and trades; Yahoo only has daily data. As an analogy, you can always make a high-definition photo more blurry and pixelated, but you can'...
chrisaycock's user avatar
  • 9,817
7 votes
Accepted

What are the proper ways to do order book downsampling?

Two chief reasons for subsampling or using a different event space are (i) computational or spatial tractability and (ii) denoising/signal extraction. Sergei's response seems to focus on the first ...
databento's user avatar
  • 2,478
6 votes

Data exported from Capital IQ, FactSet, Bloomberg, Compustat

Morningstar Morningstar partnered with Quantopian, and the latter published the structure of Morningstar's equity fundamentals database: https://www.quantopian.com/help/fundamentals Quantopian ...
Anton Tarasenko's user avatar
6 votes

Where can I find open swaption implied volatility data?

CME publishes its volatility surface daily on their FTP: CME Vol Surface. Unfortunately I know of no open APIs that would get you historical data. I'd recommend looking at Barclays Live or Morgan ...
Helin's user avatar
  • 11.6k
6 votes
Accepted

Does Yahoo/Google no longer support web-scraping of FOREX data?

Yahoo has changed their site structure. The new download URLs look like this: https://query1.finance.yahoo.com/v7/finance/download/MSFT?period1=1463461200&period2=1494910800&interval=1d&...
Brian from QuantRocket's user avatar
6 votes

Which data sources are available for cryptocurrencies?

As mentioned in other answers, Coinmarketcap and Bitcoincharts are two options. The data they provide is also available from Quandl too. The only issue is, that data has daily frequency and is only ...
Tanmay's user avatar
  • 261
6 votes
Accepted

how to treat NA values in Compustat and CRSP

You're going to have to use judgement. There are situations where treating missing values as 0 would be insane. In general, I'd be immensely cautious. On the other hand, there are situations where it'...
Matthew Gunn's user avatar
  • 6,934
6 votes

Simulating trading strategies

Is there a place online where you can simulate strategies programmatically? Your best choice is most likely a service such as Quantopian or QuantConnect. Quantopian provides equity and futures data ...
Theodore's user avatar
  • 1,172
6 votes
Accepted

Using AlphaVantage For Japan/Shanghai/Hong Kong/Shenzhen stock exchange data?

All of the examples just got tested. They all work for me with my API key. Just swap in your API key below, and you should be all set. For Tokyo: Sony: https://www.alphavantage.co/query?function=...
amdopt's user avatar
  • 4,768
6 votes

Px last in Bloomberg

What is px_last is rather complex and will depends on the terminal options installed where you downloaded it. Type DPDF to see what are your settings. You can adjust closing price to account for ...
Stravog's user avatar
  • 91
5 votes
Accepted

OTC Market Data

From what I know OTC http://www.otcmarkets.com/ has their data provided to them by EDGAR, which provides solutions such as API's which can be used to acces things like historical market data. the ...
Mustard Tiger's user avatar
5 votes
Accepted

Is futures contracts data only reported for business days?

Usually end-of-day prices refer to settlement prices; they are not the last price at which the instrument was trading during that business day. Settlement prices are important because they are the ...
SRKX's user avatar
  • 11.1k

Only top scored, non community-wiki answers of a minimum length are eligible