We’re rewarding the question askers & reputations are being recalculated! Read more.
62

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as people downvoted my answer. I certainly hope people are more open minded in the quant forum. Note - It seems that this suggestion has created some strong ...


17

As of April 2014, the 32-bit version of kdb+ is now free to try. This free version may not be used in production systems. The only technical limitation vs. the 64-bit version is that you can only address up to 4GB of memory per process.


17

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use case. If someone says they get a "x ms read", "y inserts per second", "k times speedup", "store n TB data" or "have m years of experience" and use that to ...


16

At discretelogics we just released a file format to store time series in flat files called "TeaFiles". In addition to raw data they can store the binary item layout and a description of the contents. C#, C++, Python APIs are available open source, licensed under the GPL, see discretelogics.com/teafiles/ Using memory mapping, read performance reaches that ...


14

There are many specialised products for HF tick data. In addition to KDB which you mentioned, there is OneTick, Vertica, Infobright, and some open-source ones like MonetDB etc. (see http://en.wikipedia.org/wiki/Column-oriented_DBMS). My experience is that Column Oriented Databases are overrated when it comes to tick data, because very often you request the ...


14

The reason that "traditional" NoSql databases will not get much up take in finance is that they are designed to solve a different problem. Most NoSql databases from the web world are designed with two central design parameters. First key lookups should be very fast. Second is that operations should be atomic at the row level and should not span records. This ...


14

You could look into Pandas, a Python library that integrates with PyTables. It was created by someone at AQR and has some similar features as KDB.


14

You could try Arctic. Other open source column-oriented databases that you may not have considered include LucidDB and C-Store.


13

I don't like KDB+/q. For KDB+ experts, I am not picking a fight. The following is just my own understanding on KDB+ and TimeSeries Database. You're warmly welcome to correct me if anything wrong in your eyes :). First of all, during my near one year's KDB+/q development experience, I never ever find a paper based benchmark result indicating KDB+/q ...


12

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they store data along one of the dimensions (which you must choose, usually time) contiguously on disk, and thus reads are extremely fast. The alternative, when ...


11

KDB is useful for two reasons: - Storage of data; and easy access to the data (i.e. querying ticks..etc) - Rich query language that supports many Quant functions however; what KDB does not do well; is the quant query language. I have evaluated KDB, Matlab, and R. So far R is the winner. I have not found any fast solution for storing and retrieving data; ...


8

We've created a roundup of the top column-oriented database systems: http://www.timestored.com/time-series-data/column-oriented-databases This includes kdb+ and some open source alternatives. Open: InfluxDB, Java Chronicles, OpenTSDB, KairosDB Closed: oneTick, McObject, Teradata Database, vectorwise, sybase, vertica We have done some initial work at ...


8

OpenTSDB is good for large-scale time series storage. metrilyx/opentsdb-pandas and wiktorski/opentsdb_pandas seems to provide the interface with pandas. OpenTSDB and HBase rough performance test | MoreDevs provides a benchmark, may not exactly match your requirements but you can try.


7

While noble, unfortunately, this type of effort is not very practical. Mostly because market data is a major source of revenue for the market centers and is never simply given away, at least not in intraday form. A few things to consider: Becoming a market data distributor is both costly and entails entering into agreements with each market center. If we ...


7

Have a look at Kona which is a FOSS project trying to be compatible. Also Tom Szczesny has done some work on its predecessors namely A. I hope this helps. Also if you are not looking for a perfect substitute you can have a look at other Time Series Databases like InfluxDB, Java Chronicles, OpenTSDB, KairosDB which are all Open Source. There are commercial ...


6

A columnar database or No-SQL solution may be your best choice. It depends on which OS you target, what your throughput and latency requirements are and whether you look to persist all data or not and finally how big the size of your data is expected to be. Obviously if you only look to store hourly/daily data then even a database that comprises a year of ...


6

Few points from my experience: 1 Another filters that you that you should consider is for price = 999 or 999.99 that appears in some data providers. 2 Another set of checks is to look at cross-section of e.g. range = (high-low)/close over all names. Check for the smallest range and largest range to see if the values make sense. You can also check daily % ...


6

Using MySQL for financial data is not unreasonable. But for tick data are you ever going to do anything except a query on a date range? For analyzing tick data in R I generally keep it in a disk file, one tick file per day, and load the files in as I need them. Using .RData files instead of csv files is quicker. I've also used custom C++ classes before, to ...


6

I recommend you optimize your SQL implementation instead of going for NoSQL, and throwing more expensive hardware at the problem. Always benchmark first. The reason I'm saying this is that I've seen MS SQL Server scale perfectly fine for options data of the magnitude you're describing and "big number of strikes tables will be enormously long and, hence, ...


6

Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded. I would like to recommend some new technologies and at the same time share a few of my experiences in this space. As @madilyn is trying to explain: It all depends on your use case. In my experience it's easy to know what you ...


5

As background, Floating point precision is a way of storing numbers such that the precision is relative to the largest digit. For instance, the number $0.00123$ stored in fixed precision needs 6 digits of precision (3 zeros and the 3 non-zero numbers). However, this same number stored as floating point precision $1.23 \cdot 10^{-3}$ needs only 3 ...


5

I've run into this issue as well. I can't find any official documentation, but the issue arises because the SEC makes up CUSIPs for all of the options on the list. It does this by taking the first 6 digits of the underlying equity (which makes sense, as this represents the issuer, which should be the same for the stock and the option), and then for the 7th ...


5

Disclosure: I work for the company developing ATSD. Axibase Time Series Database is not open-source but its community edition is free. Time precision is milliseconds. Value is float, double or long. EDIT 1: February 2016. ATSD JDBC Type 4 driver released under Apache 2 license to simplify data access for Java applications. EDIT 2: March 2016. Decimal ...


5

Regarding storage, I stream real-time updates for exchange listed contracts (outright + exchange listed calendar spreads) to InfluxDB. Its a time-series database, mostly geared towards IT Ops for storing log data, but it works fine with homogeneous finance data. For options strips, due to the sheer amount of data generated per day, I use TeaFiles. Pros for ...


4

mongodb seems to be a good opensource solution to store historical tick-by-tick. I am using it here and it seems to be very simple to use.


4

Have you considered the HDF5 data model? Edit for Louis : Why using HDF5 ? As stated in the HFDF short description page : HDF5 is a unique technology suite that makes possible the management of extremely large and complex data collections. HDF5 is a suitable solution when dealing with very large datasets and you need performance. Again, as stated ...


4

I know this is probably a naive answer, but when I started doing data analysis for personal trading I looked for something much faster than SQL. I program in C++ and I found that HDF5 was the answer to all my problems http://www.hdfgroup.org/HDF5/ It's not accounting oriented, but the nice thing about it is that you can do almost anything with it and it is ...


4

(P) prefix : As a service to the market and typically at the request of an issuer, Moody's will assign a provisional rating when it is highly likely that the rating will become final after all documents are received, or an obligation is issued into the market. A provisional rating is denoted by placing a (P) in front of the rating. Such ratings may also be ...


4

check this out Arctic. It's a Man AHL developed Mango DB for store their financial time series. Claimed to be really good. But i haven't try myself.


4

Typically for dynamic sources like this, you would provide A reference URL, and a date of download. For just one Quandl data series that is easy due to the Web API, i.e. in BibTeX I would do something like this: @online{QuandlXOM, author = {Quandl}, title = {{WIKI} Exxon Mobil End-Of-Day Data}, year = 2016, url = {https://www.quandl.com/data/...


Only top voted, non community-wiki answers of a minimum length are eligible