7
$\begingroup$

I'm planing to store stock market data in realtime and aggregate ticks for draw volume based cluster graph. Something like this:

enter image description here

Every tick (or second) data will be grouped by period (1,5,10 minutes; 1,4,24 hours), type (buy, sell) and price; calculated sum of volumes. Result will be something like this:

[
  {timestamp: "2016/01/30 15:04:00", period: "1m", price: 123.45, buy: 2345, sell: 1998},
  {timestamp: "2016/01/30 15:04:00", period: "1m", price: 123.46, buy: 3111, sell: 1040},
  {timestamp: "2016/01/30 15:05:00", period: "1m", price: 123.46, buy: 1421, sell: 3475},
  {timestamp: "2016/01/30 15:05:00", period: "1m", price: 123.47, buy: 6056, sell: 9138},
]

For delivery ticks from stocks to db I will use nats (https://github.com/nats-io/gnatsd). Which database I can use for store and aggregate in realtime?

$\endgroup$
4
$\begingroup$

check this out Arctic. It's a Man AHL developed Mango DB for store their financial time series. Claimed to be really good. But i haven't try myself.

$\endgroup$
  • $\begingroup$ Arctic is the python class for generation queries to MongoDB ? Something like ActiveRecord for relational databases, but special for finance? $\endgroup$ – Kroid Feb 10 '16 at 15:43
  • $\begingroup$ I think it is similar. It's a framework that you could use python to store, query your data from MongoDB. and they did many work to tune the performance. I can't be certain because I didn't use it myself. $\endgroup$ – JOHN Feb 10 '16 at 16:01
2
$\begingroup$

As @Nicholas said in a comment KX/KDB+ is popular in finance for this purpose. Direct message passing and local aggregation on the machine may be the best method in this case IMO.

$\endgroup$
  • $\begingroup$ very proprietary andneed experts to work with +q, best to go for open source tech $\endgroup$ – PirateApp Apr 14 '18 at 7:36
2
$\begingroup$

There are many different databases out there, all specialized for different use cases. The main parts you should consider are:

  1. Using a time series database, since they can handle timestamped data (e.g. ticks) more efficient than any SQL solution can by using bucketing and other methods.
  2. Using a database with a good query language, for example to aggregate multiple values, calculating highs and lows etc.

Therefore, the best solution in my opinion currently out there would be InfluxDB. Not only because the easy API for inserting and querying data, but also because of the whole stack of InfluxData.

$\endgroup$
1
$\begingroup$

Apache Cassandra would be a good fit. It's a partitioned row store, where rows are organized into table using a partition key.

It is common use case to store time series data, you could simply use ticker and a period as partition key. Cassandra is optimized for writes and it's easily scaled, but you need at least 3 servers for it to run.

$\endgroup$
  • $\begingroup$ not specifically meant for time series, influxdb or timescale are more specific for this usecase $\endgroup$ – PirateApp Apr 14 '18 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.