8
$\begingroup$

I'm planing to store stock market data in realtime and aggregate ticks for draw volume based cluster graph. Something like this:

enter image description here

Every tick (or second) data will be grouped by period (1,5,10 minutes; 1,4,24 hours), type (buy, sell) and price; calculated sum of volumes. Result will be something like this:

[
  {timestamp: "2016/01/30 15:04:00", period: "1m", price: 123.45, buy: 2345, sell: 1998},
  {timestamp: "2016/01/30 15:04:00", period: "1m", price: 123.46, buy: 3111, sell: 1040},
  {timestamp: "2016/01/30 15:05:00", period: "1m", price: 123.46, buy: 1421, sell: 3475},
  {timestamp: "2016/01/30 15:05:00", period: "1m", price: 123.47, buy: 6056, sell: 9138},
]

For delivery ticks from stocks to db I will use nats (https://github.com/nats-io/gnatsd). Which database I can use for store and aggregate in realtime?

Improve this question
$\endgroup$
2

4 Answers 4

Reset to default
5
$\begingroup$

check this out Arctic. It's a Man AHL developed Mango DB for store their financial time series. Claimed to be really good. But i haven't try myself.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Arctic is the python class for generation queries to MongoDB ? Something like ActiveRecord for relational databases, but special for finance? $\endgroup$
    – Kroid
    Commented Feb 10, 2016 at 15:43
  • $\begingroup$ I think it is similar. It's a framework that you could use python to store, query your data from MongoDB. and they did many work to tune the performance. I can't be certain because I didn't use it myself. $\endgroup$
    – JOHN
    Commented Feb 10, 2016 at 16:01
2
$\begingroup$

As @Nicholas said in a comment KX/KDB+ is popular in finance for this purpose. Direct message passing and local aggregation on the machine may be the best method in this case IMO.

Improve this answer
$\endgroup$
2
  • $\begingroup$ very proprietary andneed experts to work with +q, best to go for open source tech $\endgroup$
    – PirateApp
    Commented Apr 14, 2018 at 7:36
  • $\begingroup$ KDB+ is extremely expensive - 6 figures per year and up. They do have a free version, but it's time-limited and strictly for non-commercial use. Any revenue generating activity if forbidden. $\endgroup$
    – Tullochgorum
    Commented Dec 11, 2019 at 20:06
2
$\begingroup$

There are many different databases out there, all specialized for different use cases. The main parts you should consider are:

  1. Using a time series database, since they can handle timestamped data (e.g. ticks) more efficient than any SQL solution can by using bucketing and other methods.
  2. Using a database with a good query language, for example to aggregate multiple values, calculating highs and lows etc.

Therefore, the best solution in my opinion currently out there would be InfluxDB. Not only because the easy API for inserting and querying data, but also because of the whole stack of InfluxData.

Improve this answer
$\endgroup$
1
$\begingroup$

Apache Cassandra would be a good fit. It's a partitioned row store, where rows are organized into table using a partition key.

It is common use case to store time series data, you could simply use ticker and a period as partition key. Cassandra is optimized for writes and it's easily scaled, but you need at least 3 servers for it to run.

Improve this answer
$\endgroup$
1
  • $\begingroup$ not specifically meant for time series, influxdb or timescale are more specific for this usecase $\endgroup$
    – PirateApp
    Commented Apr 14, 2018 at 7:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.