101

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as people downvoted my answer. I certainly hope people are more open minded in the quant forum. Note - It seems that this suggestion has created some strong ...


34

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use case. If someone says they get a "x ms read", "y inserts per second", "k times speedup", "store n TB data" or "have m years of experience" and use that to ...


19

As of April 2014, the 32-bit version of kdb+ is now free to try. This free version may not be used in production systems. The only technical limitation vs. the 64-bit version is that you can only address up to 4GB of memory per process.


17

If the only purpose is to backtest with the data, the primary (in some cases, only) access pattern is to seek to a start time and read all of the data serially through to an end time. Then, there is a strong argument for storing it in plain, flat files with binary encoding, i.e. dumping the data structs in their in-memory layout straight to disk. Storing it ...


15

I don't like KDB+/q. For KDB+ experts, I am not picking a fight. The following is just my own understanding on KDB+ and TimeSeries Database. You're warmly welcome to correct me if anything wrong in your eyes :). First of all, during my near one year's KDB+/q development experience, I never ever find a paper based benchmark result indicating KDB+/q ...


15

You could try Arctic. Other open source column-oriented databases that you may not have considered include LucidDB and C-Store.


15

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they store data along one of the dimensions (which you must choose, usually time) contiguously on disk, and thus reads are extremely fast. The alternative, when ...


15

Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded. I would like to recommend some new technologies and at the same time share a few of my experiences in this space. As @madilyn is trying to explain: It all depends on your use case. In my experience it's easy to know what you ...


9

We've created a roundup of the top column-oriented database systems: http://www.timestored.com/time-series-data/column-oriented-databases This includes kdb+ and some open source alternatives. Open: InfluxDB, Java Chronicles, OpenTSDB, KairosDB Closed: oneTick, McObject, Teradata Database, vectorwise, sybase, vertica We have done some initial work at ...


9

OpenTSDB is good for large-scale time series storage. metrilyx/opentsdb-pandas and wiktorski/opentsdb_pandas seems to provide the interface with pandas. OpenTSDB and HBase rough performance test | MoreDevs provides a benchmark, may not exactly match your requirements but you can try.


8

Have a look at Kona which is a FOSS project trying to be compatible. Also Tom Szczesny has done some work on its predecessors namely A. I hope this helps. Also if you are not looking for a perfect substitute you can have a look at other Time Series Databases like InfluxDB, Java Chronicles, OpenTSDB, KairosDB which are all Open Source. There are commercial ...


7

An SQL database is generally best for structured data, ad-hoc queries and for queries involving joining several entities together to find the results. It will also help you maintain data consistency and integrity by forcing this more structured design. Recent in-memory features of modern database engines offer most of the remaining performance advantages of ...


6

You're going to have to use judgement. There are situations where treating missing values as 0 would be insane. In general, I'd be immensely cautious. On the other hand, there are situations where it's reasonable to attempt an alternative calculation and situations where using 0 may be reasonable. A nice example can be found in Kenneth French's description ...


5

Have you considered the HDF5 data model? Edit for Louis : Why using HDF5 ? As stated in the HFDF short description page : HDF5 is a unique technology suite that makes possible the management of extremely large and complex data collections. HDF5 is a suitable solution when dealing with very large datasets and you need performance. Again, as stated ...


5

Cassandra is the obvious choice. With MongoDB or any RDBMS, you will hold all ticks in a table (collection in Mongo-speak) and index by ticker. This means that when you want to retrieve data for a ticker, the data will not be contiguously stored, and you will have a massive usage of index and random reads. Even with SSDs this is slow. For 500k ticks into ...


5

At the risk of reopening an old question, I thought that I would offer my experience. I worked for a competitor of Man AHL (who created artic). We used a columnar database called HP Vertica. Its not free unfortunately. We used it as a huge time series database for many use cases. We had one cluster of 3 fairly powerful machines that gave us redundancy if ...


5

As background, Floating point precision is a way of storing numbers such that the precision is relative to the largest digit. For instance, the number $0.00123$ stored in fixed precision needs 6 digits of precision (3 zeros and the 3 non-zero numbers). However, this same number stored as floating point precision $1.23 \cdot 10^{-3}$ needs only 3 ...


5

I've run into this issue as well. I can't find any official documentation, but the issue arises because the SEC makes up CUSIPs for all of the options on the list. It does this by taking the first 6 digits of the underlying equity (which makes sense, as this represents the issuer, which should be the same for the stock and the option), and then for the 7th ...


5

(I work for Axibase) Axibase Time Series Database is not open-source but it's free on single node. Time precision is microseconds. It supports OLCHV+VWAP aggregators in SQL and REST API with various functions for filtering by trading calendar, auction stages, indices etc. SELECT datetime, symbol, close(), vwap() FROM atsd_trade WHERE in_index('<index-...


5

check this out Arctic. It's a Man AHL developed Mango DB for store their financial time series. Claimed to be really good. But i haven't try myself.


5

Regarding storage, I stream real-time updates for exchange listed contracts (outright + exchange listed calendar spreads) to InfluxDB. Its a time-series database, mostly geared towards IT Ops for storing log data, but it works fine with homogeneous finance data. For options strips, due to the sheer amount of data generated per day, I use TeaFiles. Pros for ...


4

(P) prefix : As a service to the market and typically at the request of an issuer, Moody's will assign a provisional rating when it is highly likely that the rating will become final after all documents are received, or an obligation is issued into the market. A provisional rating is denoted by placing a (P) in front of the rating. Such ratings may also be ...


4

The only thing which comes close to kdb in my opinion is QuestDB. They are one of the few projects as a TSDB which have speed as a priority. They recently added out-of-order inserts which allowed them to benchmark agains some of the other TSDBs out there and the results were a quite impressive 1.4 millions writes per second (source). I have been using it (...


4

You're going to want different databases for different data. For instance, the company master, historical data, and fundamental data can probably all live in a standard SQL database (MySQL or Postgres are both reasonable choices). If the intraday data is relatively low-frequency (e.g., 1-minute bars or lower), that can probably be put into the SQL DB as ...


4

Typically for dynamic sources like this, you would provide A reference URL, and a date of download. For just one Quandl data series that is easy due to the Web API, i.e. in BibTeX I would do something like this: @online{QuandlXOM, author = {Quandl}, title = {{WIKI} Exxon Mobil End-Of-Day Data}, year = 2016, url = {https://www.quandl.com/data/...


4

As always, any answer to this question is hugely driven by your use cases. The following is my interpretation and ansatz. Here on QSE and over at the DB exchange, I have seen various approaches to keeping financial / market data - and the corresponding discussions are quite heated... Some suggest simply dumping/reading to/from CSV, others suggest time series ...


3

Github and Bitbucket are both good options for a git repository. If you want other people to look at your code and help build on it, Github is the better choice. If privacy is what matters most, you can get a free private repository from Bitbucket. On github, only public repositories are free.


3

Recent benchmarks of KDB+ vs. other big data technologies, Kdb still comes out on top 5, especially considering hardware costs for this particular data set. [1.1 Billion Taxi Rides on kdb+/q & 4 Xeon Phi CPUs] http://tech.marksblogg.com/billion-nyc-taxi-kdb.html


3

I am wondering if anyone has used NoSQL ... to store and analyze data. Yes. Have a look at arctic on github. This is an open-source API built on top of MongoDB, that is in production use by one of the largest hedge funds in the world, for storing time-series data. I would imagine that NoSQL would be much faster. In the github wiki you'll find links to ...


3

Lots of people focus on data storage ability and compare KDB with other SQL/query-based databases. Such comparison is like considering "Is a Ferrari good for running a bus route?" KDB+ is capable of manipulating and querying large data set. The performance is fast (in comparison to most RDBMS) due to its column based storage, but it's not her strongest suit....


Only top voted, non community-wiki answers of a minimum length are eligible