I've just found out about this forum while searching for answers for an InfluxDB performance "issue". I'm using it to store financial tick data (7 fields per row), query it and process it into candles. At the moment I'm querying around 1 billion ticks, about 4 years of data, for a total weight of about 28GB. The first problem was InfluxDB loads the entire query on Ram every time, so my computer would crash; I had to solve this by subdividing the query into chunks manually (month by month), and iterating through them while appending processed data to a DataFrame, exporting it at the end of the last query. When I got it to work, those were my query times (number of rows on the left, query time in hours, minutes and seconds in the center, processing time for creating the candles, on the right):
9480222, 2:07:46, 38:26
12839124, 3:06:02, 24:48
17256737, 4:19:54, 26:07
13716707, 3:28:37, 18:54
12671435, 2:35:27, 20:58
11112483, 2:15:53, 26:53
17055181, 3:34:21, 52:18
21232810, 6:29:42, 1:33:52
16935780, 4:47:56, 1:05:11
Those numbers seem a bit off. The average is around 60-70k rows per minute, 1k rows per second. Since this is my first experience with TS Databases and with Influx, would you consider this performance normal? I'm running InfluxDB 1.7.9, Influx Python client 5.2.3, Python 3.7, running from Pycharm, on a MacBook Pro with 16GB Ram.
Any tip is welcomed!
I also read posts were a lot of people were advising on not using a database at all, just chunking data and storing it in files. That's something I intended to do for heavier data such as Order Book data, but I've already spent quite some time setting this up and building my infrastructure around it and I'd like to make it work decently at least for the time being.
Thanks in advance
