Tag Info

Check out http://discretelogics.com/teafiles/. It solves the large memory footprint by using memory mapped files.

I recommend you optimize your SQL implementation instead of going for NoSQL, and throwing more expensive hardware at the problem. Always benchmark first. The reason I'm saying this is that I've seen MS SQL Server scale perfectly fine for options data of the magnitude you're describing and "big number of strikes tables will be enormously long and, hence, ...

You need to log some data, and later use it for analysis. Dare I suggest you just append the values to a set of files? You can load up the data later for your runs, and cache anything needed frequently. Frankly buying an SSD and copying the files to that before you run analysis should solve your problem, and no-one had to get hurt.

A columnar database or No-SQL solution may be your best choice. It depends on which OS you target, what your throughput and latency requirements are and whether you look to persist all data or not and finally how big the size of your data is expected to be. Obviously if you only look to store hourly/daily data then even a database that comprises a year of ...