We would like to store financial tick data in a database (potentially billions of rows) and then create aggregated (open-high-low-close) bar data from it (e.g. 1min or 5min bars).
It was mentioned to us that a NoSQL or time series database might be a good choice for this. Can anybody give any advice on which open source product might fit this requirement best.
Note: query performance is very important for us.
In our research we came across the following products (maybe there are more):
We did run a test with InfluxDB with around 10 million ticks. Unfortunately the creation of 1min bars was 3-5 slower than with a relation database (i.e. MySQL).
We are aware that KDB now offers a free 32-bit version, but unfortunately 32-bit will not be enough for our use case.
Any advice is appreciated.
EDIT (Sept 2015): We also did a test with OpenTSDB which seems to be quite fast. The import of 10 mio. prices took about one minute and the aggregation into 1 Min Bars took about 5 seconds.
EDIT (Jan 2017): More than one year after the initial test we gave InfluxDB another try and it turns out that they have made huge progress in the meantime. Write performance is now up to 2 mio. data points per second (with version 1.2)! We have now decided to integrate InfluxDB into our own product AlgoTrader