Skip to main content
108 votes
Accepted

Building Financial Data Time Series Database from scratch

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as ...
drobertson's user avatar
  • 1,882
35 votes

Building Financial Data Time Series Database from scratch

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use ...
madilyn's user avatar
  • 5,240
17 votes
Accepted

What's the most efficient way to store options and time series data for backtesting?

If the only purpose is to backtest with the data, the primary (in some cases, only) access pattern is to seek to a start time and read all of the data serially through to an end time. Then, there is a ...
databento's user avatar
  • 2,508
16 votes

Building Financial Data Time Series Database from scratch

Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded. I would like to recommend some ...
chjortlund's user avatar
15 votes

Building Financial Data Time Series Database from scratch

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they ...
Thomas Browne's user avatar
10 votes
Accepted

Efficient way to store orderbook in Python

The best solution is to change how you're doing it completely: Store the data before it even gets to your program and process it later. e.g. Use your program merely as a daemon to subscribe to the ...
databento's user avatar
  • 2,508
7 votes

Which Database (MySql or NoSQL) for a Stock market App

An SQL database is generally best for structured data, ad-hoc queries and for queries involving joining several entities together to find the results. It will also help you maintain data consistency ...
Finn Espen Gundersen's user avatar
6 votes

Is there any thing out there as a substitute for KDB?

The only thing which comes close to kdb in my opinion is QuestDB. They are one of the few projects as a TSDB which have speed as a priority. They recently added out-of-order inserts which allowed them ...
Newskooler's user avatar
6 votes

Is there any thing out there as a substitute for KDB?

At the risk of reopening an old question, I thought that I would offer my experience. I worked for a competitor of Man AHL (who created artic). We used a columnar database called HP Vertica. Its not ...
ThatDataGuy's user avatar
6 votes
Accepted

how to treat NA values in Compustat and CRSP

You're going to have to use judgement. There are situations where treating missing values as 0 would be insane. In general, I'd be immensely cautious. On the other hand, there are situations where it'...
Matthew Gunn's user avatar
  • 6,974
5 votes

Building Financial Data Time Series Database from scratch

Regarding storage, I stream real-time updates for exchange listed contracts (outright + exchange listed calendar spreads) to InfluxDB. Its a time-series database, mostly geared towards IT Ops for ...
omencat's user avatar
  • 161
5 votes
Accepted

Backtesting vs live trading data handling and abstraction

Yes, I recommend making historical backtests and live trading as similar as possible. This leaves you one lesser source of variability when you inevitably see different backtest and live results. Do ...
databento's user avatar
  • 2,508
4 votes

Is there any thing out there as a substitute for KDB?

Recent benchmarks of KDB+ vs. other big data technologies, Kdb still comes out on top 5, especially considering hardware costs for this particular data set. [1.1 Billion Taxi Rides on kdb+/q & 4 ...
Ari's user avatar
  • 81
4 votes

Which Database (MySql or NoSQL) for a Stock market App

You're going to want different databases for different data. For instance, the company master, historical data, and fundamental data can probably all live in a standard SQL database (MySQL or Postgres ...
Thomas Johnson's user avatar
4 votes
Accepted

How to cite quandl data

Typically for dynamic sources like this, you would provide A reference URL, and a date of download. For just one Quandl data series that is easy due to the Web API, i.e. in BibTeX I would do ...
Brian B's user avatar
  • 14.9k
4 votes
Accepted

suggestions for improving monitoring of trading bots?

There's no right way to do this. Some universal tips I can give are: Operational vs strategy monitoring On the first pass, you need to distinguish between these two as the tools for operational and ...
madilyn's user avatar
  • 5,240
4 votes

How to structure a stock market data database

As always, any answer to this question is hugely driven by your use cases. The following is my interpretation and ansatz. Here on QSE and over at the DB exchange, I have seen various approaches to ...
Kermittfrog's user avatar
  • 6,772
4 votes
Accepted

Where to find dissertations in risk management

The following schools have open databases for dissertations, with topics related to risk management: Princeton Operations Research and Financial Engineering Department Example: A New Generation of ...
William Wu's user avatar
3 votes

Where can I find CMS swap trading prices?

The spread are quoted on ICAP or in Bloomberg if you have acess to them or you can refer to the paper of Mercurio where you have some quotes and examples
Bond007's user avatar
  • 83
3 votes

Usage of NoSQL storage in Finance

I am wondering if anyone has used NoSQL ... to store and analyze data. Yes. Have a look at arctic on github. This is an open-source API built on top of MongoDB, that is in production use by one of ...
Luciano's user avatar
  • 151
2 votes

Is there any thing out there as a substitute for KDB?

If you're looking for a columnar store alternative, have a look at Druid.io Druid is a good fit if you are after 1. fast aggregations and searches 2. Real-time analysis 3. Huge amount of data (...
Noel's user avatar
  • 121
2 votes

Is there any thing out there as a substitute for KDB?

man-group/ArcticDB: High performance datastore for time series and tick data seems a good alternative for time series and tick data. It is built upon mongodb.
xgdgsc's user avatar
  • 246
2 votes

Is there any thing out there as a substitute for KDB?

If you need the column-store for OLAP use cases (not the time series, but this could be done too), try Clickhouse, for 1.1 billion taxi benchmark, it comes 2nd after kx kdb+, and the fastest if ...
Kokizzu's user avatar
  • 121
2 votes

Is there any thing out there as a substitute for KDB?

We used Vertica at my last job at a large proprietary trading firm with hundreds of employees.
Katie's user avatar
  • 304
2 votes

Is there any thing out there as a substitute for KDB?

If you absolutely need a DBMS, the two obvious candidates are Clickhouse and QuestDB, which others have mentioned. However, there are good chances that you don't need a DBMS. In which case, a plain ...
databento's user avatar
  • 2,508
2 votes

S&P's Sovereign Ratings: Clarification on Definitions and Symbols

Credit rating agencies all disclose their codes (and a general methodology) online. "The 'u' identifier and 'unsolicited' designation are assigned to credit ratings initiated by parties other than ...
Peacock's user avatar
  • 196
2 votes

Which database to choose for storing and aggregating finance data?

There are many different databases out there, all specialized for different use cases. The main parts you should consider are: Using a time series database, since they can handle timestamped data (e....
Martin Seeler's user avatar
2 votes
Accepted

Where can I find CMS swap trading prices?

From On Valuing Constant Maturity Swap Spread Derivatives "The CMS tickers are represented as USSWAPyy, where yy is the year indicator. For Example the tickers for CMS 30 yrs and CMS 2 yrs are ...
Ted Taylor of Life's user avatar
2 votes

Creating Options Database

I don't have much experience with database designing so I am not sure how to proceed. Read an introductory database book! Which database will be best to use? MSSQL, MySQL, NoSQL, etc.? MySQL and ...
madilyn's user avatar
  • 5,240
2 votes

Building Financial Data Time Series Database from scratch

I will prefix my answer with the following: I've not read the holy wars stuff in the other answers in detail. I think it's missing the point. I've implemented exactly this project more than once in ...
ThatDataGuy's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible