Timeline for Efficiently storing real-time intraday data in an application agnostic way
Current License: CC BY-SA 3.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 19, 2018 at 16:09 | comment | added | PirateApp | what is the right data type to store OHLC volume values and indicator values (RSI moving average, stochastics, bollinger etc etc) | |
| Apr 11, 2011 at 11:35 | vote | accept | Karol J. Piczak | ||
| Apr 9, 2011 at 12:25 | comment | added | Karol J. Piczak | I see, indeed I found later yesterday that with Kdb+ it was an implementation decision (append only) to maintain performance. Infobright also has limitations on DML operations in its free version. But I don't like it. It's like taking away a knife from your kitchen so that you accidentally don't get hurt. InfiniDB approach seems more relaxed here. Full DML is supported. And it's left to the user to know the impact. Sure, inserting something at the front of a column is essentially equal to rewriting it from scratch. But randomly updating non-sorted values shouldn't be a problem. | |
| Apr 8, 2011 at 19:56 | comment | added | chrisaycock | As for bitemporal, feel free to rewrite your entire OHLC table if you want, though really you should only need to append the last day's worth of market data anyway since raw prices don't change. If your concern is adjusted prices (ie, corporate actions), I'm of the belief that you should only store raw data and then pre-compute the adjusted on-the-fly. To do this, simply store splits, dividends, and name changes in a separate table, then provide a user-level function to materialize the adjusted data. | |
| Apr 8, 2011 at 19:51 | comment | added | chrisaycock | @Karol On-disk updates are impossible from an implementation standpoint. Columns are stored as files (each column is its own file of sequential data, so it can be read into memory as an array). So any (non-append) update requires rewriting the entire file; this is the exact process your text editor does every time you change some code. That means you'll need to reset any reading process that may have memory-mapped the column prior to update to ensure consistency (as per CAP Theorem). Kdb+ doesn't support reseting readers natively; a competing product might, but it's still a nasty process. | |
| Apr 8, 2011 at 19:02 | comment | added | Karol J. Piczak | Thank you for a thorough answer. That's exactly what I'm trying to do - learn more so that I know what are the pros and cons of available options. I'm not sure I understood the penultimate section though. You mention that on-disk update operations are practically impossible. You mean from a performance standpoint not implementation constraints, right? Something like worst case being roughly O(n)? And about bitemporal - if there are no compliance requirements, isn't it an overkill artificially blowing up the amount of data if for instance all I need is close price adjustment for stock splits? | |
| Apr 8, 2011 at 18:00 | history | answered | chrisaycock | CC BY-SA 3.0 |