Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded.

I would like to recommend some new technologies and at the same time share a few of my experiences in this space.

As @madilyn is trying to explain: It all depends on your use case. In my experience it's easy to know what you want to do today, but really hard to foresee all the future use cases. Therefore I also take the agility into account when making a choice of the stack to use for a given system.

Flat files is just really powerful especially if they are stored in a binary format (e.g. tea, HDF5, Feather, Apache Parquet), using JSON and serializing / de-serializing the data is not clever at all.

When dealing with huge amounts of rows of structured data a modern Column-oriented database is hard to beat, especially in combination with interesting technologies like snappy (compression that... well compress AND gives faster i/o - What!?!) and distributed file systems (e.g. GlusterFS, GridFS, CEPH), that allows you to build a relative inexpensive and scalable database cluster, check out MariaDB Columnstore and the super performant (but with some drawbacks) Clickhouse.

Most of the data in finance is having the time dimension, so it might be a good idea of thinking this into the? KDB+ has been around for decades with a super strong database, unfortunately mostly unavailable for smaller companies due to cost. Now a whole sub-industry is emerging, fueled by the IoT buzz, offering time series databases (e.g. InfluxDB, RiakTS, OpenTSDB), but in my opinion the latest and still relative unknown contender TimescaleDB offers some truly unique features. TimescaleDB is an extension to PostgreSQL and offers time series capabilities inside the same database of where your non-time-dimensional data resides, making it easy to JOIN together and in general leveraging PostgreSQL's large feature set.

The query language is really important, and in my opinion nothing beats SQL (and NewSQL) in terms of compatibility. By forcing other people to learn and use CQL or MongoDB Query Language, you might very well end up building a data graveyard.

When NoSQL was at the peak of its hype, I was one of the cool kids onboard with a MongoDB database, quickly I realized that a. I was the only user b. I wanted to use BI tools for the early data exploration, MongoDB have properly had a lot of customers with the same request, so luckily they have made a 'Connector for BI'. I tried both the official one and a bunch of the SQL connectors made by 3rd party vendors, and let me just say this: You don't have to do the same experiment, unless you wish to waste a day or two of your life.

Conclusion I will (party) side with @madilyn PostgreSQL with TimescaleDB extension might be the way to go for the OP, but if you’re not already married to PostgreSQL then also checkout MariaDB Columnstore, once you have a nice solution, then build a feature/script to extract data into a binary file for one-off / event research.

Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded.

I would like to recommend some new technologies and at the same time share a few of my experiences in this space.

As @madilyn is trying to explain: It all depends on your use case. In my experience it's easy to know what you want to do today, but really hard to foresee all the future use cases. Therefore I also take the agility into account when making a choice of the stack to use for a given system.

Flat files is just really powerful especially if they are stored in a binary format (e.g. tea, HDF5, Feather, Apache Parquet), using JSON and serializing / de-serializing the data is not clever at all.

When dealing with huge amounts of rows of structured data a modern Column-oriented database is hard to beat, especially in combination with interesting technologies like snappy (compression that... well compress AND gives faster i/o - What!?!) and distributed file systems (e.g. GlusterFS, GridFS, CEPH), that allows you to build a relative inexpensive and scalable database cluster, check out MariaDB Columnstore and the super performant (but with some drawbacks) Clickhouse.

Most of the data in finance is having the time dimension, so it might be a good idea of thinking this into the? KDB+ has been around for decades with a super strong database, unfortunately mostly unavailable for smaller companies due to cost. Now a whole sub-industry is emerging, fueled by the IoT buzz, offering time series databases (e.g. InfluxDB, RiakTS, OpenTSDB), but in my opinion the latest and still relative unknown contender TimescaleDB offers some truly unique features. TimescaleDB is an extension to PostgreSQL and offers time series capabilities inside the same database of where your non-time-dimensional data resides, making it easy to JOIN together and in general leveraging PostgreSQL's large feature set.

The query language is really important, and in my opinion nothing beats SQL (and NewSQL) in terms of compatibility. By forcing other people to learn and use CQL or MongoDB Query Language, you might very well end up building a data graveyard.

When NoSQL was at the peak of its hype, I was one of the cool kids onboard with a MongoDB database, quickly I realized that a. I was the only user b. I wanted to use BI tools for the early data exploration, MongoDB have properly had a lot of customers with the same request, so luckily they have made a 'Connector for BI'. I tried both the official one and a bunch of the SQL connectors made by 3rd party vendors, and let me just say this: You don't have to do the same experiment, unless you wish to waste a day or two of your life.

Conclusion I will (partly) side with @madilyn PostgreSQL with TimescaleDB extension might be the way to go for the OP, but if you’re not already married to PostgreSQL then also checkout MariaDB Columnstore, once you have a nice solution, then build a feature/script to extract data into a binary file for one-off / event research.