Reputation
737
Top tag
Next privilege 750 Rep.
See votes, expandable usercard
Badges
1 7 17
Impact
~43k people reached

Aug
14
suggested approved edit on portfolio optimisation with VaR (or CVaR) constraints
Aug
5
comment Why use a column database for tick/bar data?
@JL344 Re the "Row-oriented databases just aren't designed to handle a huge number of tables as effectively as a huge number of rows in one table." comment, it seems this is a good point. MySQL uses one file per table, so it should be fairly unlimited. However see (serverfault.com/q/83438/87322) where meta queries can become notably slow once you have thousands of tables, apparently. (This is a MySQL bug that is fixable, rather than a flaw with the row-oriented concept.)
Aug
5
comment Why use a column database for tick/bar data?
@JL344 One table per symbol of OHLCV data in a row-oriented DB is very different from the column-oriented DB. In the former they are are stored in row order; in the latter they are stored in column order. In the former appending a new row requires appending, say, 44 bytes, to a single file; in the later it requires appending 8-12 bytes to each of 5 separate disk files. That is their advantage; the gist of Q1 and Q2 is for this usage scenario do column-oriented DBs bring any other advantage?
Aug
5
comment Why use a column database for tick/bar data?
@JL344 You didn't mention the row-oriented approach of one table per symbol (see the UPDATE in my question). Is there a reason people are not using that approach?
Aug
3
comment Why use a column database for tick/bar data?
@Freddy Thanks for the update, sorry I only just saw it. It sounds like you are confusing NoSQL DBs (which, to me, means MongoDB, CouchDB, Cassandra, Redis, etc.) with column-oriented databases? Or do you regard them all as NoSQL solutions?
Jul
17
comment Are there any standard techniques for adding realistic synthetic microstructure noise to a price series?
@Serg Is this (using the difference between future and current) any better than using the difference between previous and current? I.e. your answer implies that if the market is about to get more volatile (bigger jumps between prices) then there is more noise in current. Is that an observable phenomena?
Jul
10
comment Why use a column database for tick/bar data?
@chrisaycock By "aggregate" do you mean things like turning ticks into 1m bars, 1m bars into hourly bars, etc.? And/or do you mean making moving averages and other more complicated indicators? And/or something else?
Jul
10
comment Why use a column database for tick/bar data?
Thanks @chrisaycock I had read one of those in my hunt yesterday, but the first link I'd missed and it was very informative (I'm still working through the linked 85-page PDF, but that looks useful too). I've added more information to my question to explain why I don't feel the linked-to answers fully answer my question.
Jul
10
revised Why use a column database for tick/bar data?
Explain why I don't think other similars questions are answering the question here.
Jul
10
revised Efficiently storing real-time intraday data in an application agnostic way
edited tags
Jul
9
comment Why use a column database for tick/bar data?
Another question: I usually think in terms of one table per symbol (instrument/contract). Reading between the lines of your answer, when you have a column-oriented DB do you keep all symbols in one table?
Jul
9
comment Why use a column database for tick/bar data?
Thanks for the reply Freddy. When you say column DBs are all about Key/Value, what is the key? When I think tick/bar data, the key is a datestamp, but that is a row-oriented concept, isn't it?
Jul
9
asked Why use a column database for tick/bar data?
Jul
9
awarded  Organizer
Jul
9
revised Usage of NoSQL storage in Finance
edited tags
Jun
21
answered Tick data collection
May
30
revised Reseach on when people/institutions sell?
I'm guessing "noble prices" was supposed to be "Nobel prizes"
May
30
suggested approved edit on Reseach on when people/institutions sell?
May
26
awarded  Popular Question
Apr
18
accepted How to remove the risk element from a set of fixed rate mortgage offerings?