Timeline for Best way to store hourly/daily options data for research purposes
Current License: CC BY-SA 3.0
23 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 30, 2016 at 4:27 | comment | added | sashkello | @Cuedrah Retrospectively, I'm happy with my choice, and feel like for my purposes +'s outweigh -'s, but there were a few moments when I seriously was thinking I should migrate the whole thing... | |
| Apr 30, 2016 at 4:25 | comment | added | sashkello | @Cuedrah Among the negatives, it's a bit steeper learning curve than SQL. Aggregation is not very easy to understand, indexing is very important and not that straightforward either. One of the things hard to get past is the fact that in SQL you just have a table in easily readable format you can browse through. Not so much for mongo, you need to first aggregate a library of functions in programming language of your choice, which will display data in an easily readable manner. As soon as you have that, it's easier to manage... Otherwise, at first it might get quite annoying. | |
| Apr 30, 2016 at 4:18 | comment | added | sashkello | @Cuedrah I ended up with mongodb. The big benefit for me is that it's quite flexible, and so for someone not well versed in data management and have to reorganize and reshuffle stuff often (every time I understand I did things wrong), nosql is much better in that sense. I use flat structure with no arrays within documents or anything like that, so lots of duplicate information which is not good for space, but very good for quick reads. As soon as you feel comfortable with aggregate queries, summarizing and grouping data works awesome as well. | |
| Apr 29, 2016 at 21:26 | comment | added | Cuedrah | Just wondering what you ended up doing and how it worked out for you. I'm in the same place as you were when you asked this question and would really like to hear about your experience. | |
| Nov 6, 2014 at 14:42 | history | protected | CommunityBot | ||
| Oct 29, 2014 at 19:05 | answer | added | Thomas Browne | timeline score: 1 | |
| Oct 29, 2014 at 7:27 | comment | added | Arun Raja | @sashkello I am planning to use windows machine. So please can you pass me the details of the database design and installation details. I would be writing the data once or twice per day. Can you please mail the details at [email protected] | |
| Oct 28, 2014 at 13:13 | comment | added | sashkello | @ArunRaja Ended up with mongodb, since I don't care about writing speed, only reading speed. So, just raw timestamped data with a couple of proper indexes did it for me. | |
| Oct 24, 2014 at 4:31 | comment | added | Arun Raja | @sashkello Have you come with the database schema and model. What have you used. Have you used SQL or NoSQL? I also have a requirement quite same to your requirement. | |
| Jun 26, 2013 at 23:52 | vote | accept | sashkello | ||
| Jun 4, 2013 at 2:36 | comment | added | TomTom | "Enourmously long"? My execution table in my backtest archive had - we just wiped it due to some code issues we found - 2.5 billion rows, and is expected to grow. We collect in db form minute price data (for fast charts) and market profile per hour. I would not call that "a lot of data". Putting in a couple of SSD is cheap - and gives you HUGH read speed. What exactly is the problem you have here? | |
| May 9, 2013 at 15:24 | answer | added | unclepaul84 | timeline score: 3 | |
| May 9, 2013 at 3:53 | comment | added | Matt Wolf | @sashkello, I recommend you to think only about your requirements first. Do not get confused by someone who is ecstatic about Redis or SQL or what have you. You want to store data (speed is not so important), you want to query data fast and flexibly, you probably want to query it in R as well because you mentioned you want to profile and analyze said data. You want to look for a solution that can grow dynamically and which is extensible. Read up what people use to store time series. After that decide whether any SQL solution actually makes sense here or other solutions solve the problem better | |
| May 8, 2013 at 19:56 | history | tweeted | twitter.com/#!/StackQuant/status/332222519719378946 | ||
| May 8, 2013 at 19:54 | answer | added | madilyn | timeline score: 7 | |
| May 8, 2013 at 11:28 | comment | added | sashkello | Yep, I've seen it. But that's why I'm differentiating from similar questions because I don't need tick data and so don't need quick writing. Also it seems that it is mostly about flat time series storage, but I want to have DB designed to fit options data particularly well. | |
| May 8, 2013 at 11:23 | comment | added | chrisaycock | Have you seen this earlier question? The topic of tick storage comes-up a lot on here. | |
| May 8, 2013 at 8:21 | answer | added | Phil H | timeline score: 3 | |
| May 8, 2013 at 7:23 | history | edited | sashkello | CC BY-SA 3.0 |
added 262 characters in body
|
| May 8, 2013 at 7:13 | history | edited | sashkello | CC BY-SA 3.0 |
added 262 characters in body
|
| May 8, 2013 at 6:23 | answer | added | Matt Wolf | timeline score: 6 | |
| May 8, 2013 at 3:53 | review | First posts | |||
| May 8, 2013 at 5:05 | |||||
| May 8, 2013 at 3:35 | history | asked | sashkello | CC BY-SA 3.0 |