Tell me more ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.
up vote 13 down vote favorite
9

I am wondering if anyone has used NoSQL (mongodb, cassandra, etc.) to store and analyze data. I tried searching the web but was not able to see if the financial firms had gotten in to using nosql storage.

So far even on this site i have only seen talks of using SQL DBs. I would imagine that NoSQL would be much faster.

Would someone shed light on the subject of NoSQL solutions in the finance world. have you seen/heard anyone use it? if so why are they using. if not using then why not.

share|improve this question

4 Answers

up vote 16 down vote

Specialised NoSQL database systems are used a lot for time series storage, particularly for tick data:

  • Kx / Kdb is one prominent solution; from Arthur Whitney et al who did A+ at Morgan Stanley
  • Onetick is another, newer entrant, which has traces back to Goldman Sachs
  • Voltdb is something by serial database inventer Michael Stonebreaker
  • SciDB is another recent Stonebreaker project, trying 'commercial open source' but more for science apps (at least for now?)

The commercial offerings tend to be rather expensive and I have not heard of use of standard Web 2.0 NoSQL databases in trading environments. Some may of course do rather than talk ...

share|improve this answer
thank you this is very helpful. i have used nosql and love it (used to only serve data for search engines for enterprises). the data is still stored in DB in the backend. – user697697 Jul 1 '11 at 20:35
Kdb+ and OneTick are column-oriented stores, which is a frequent topic here. Stonebreaker's own column-oriented DB is Vertica, which was recently bought by HP. (VoltDB is row-oriented and intended for OLTP applications, whereas the column-oriented products here are intended for OLAP.) – chrisaycock♦ Jul 2 '11 at 3:26
Chris, thanks for expanding the answer.. Besides Vertica, one could also mentions Sybase's offering in the area. I do think SciDB is column-oriented too, though, and could become a dark horse. They talk R support right from the start. – Dirk Eddelbuettel Jul 2 '11 at 7:40
@Dirk Yes, Sybase IQ (now part of SAP) is also in this space. There's Vhayu (part of Thomson Reuters) too, though I haven't heard much from them in a while. I'm still confused as to what SciDB actually does. – chrisaycock♦ Jul 2 '11 at 15:21
Note that the link for SciDB should be www.scidb.org, not .com. – joshayers Dec 13 '11 at 21:20
show 1 more comment
up vote 7 down vote

The reason that "traditional" NoSql databases will not get much up take in finance is that they are designed to solve a different problem. Most NoSql databases from the web world are designed with two central design parameters. First key lookups should be very fast. Second is that operations should be atomic at the row level and should not span records. This allows the database to be sharded very effectively since no operation should span multiple machines and in turn this allows them to build scale out architectures. These databases are built for very write heavy workloads and to be able to have fast key lookups to render webpages. While it is possible to use them to store tick data its not really what they are good at.

Financial data tends to be small relative to large web companies data. So scale out in finance is less important. Also financial data tends to partition well (at the day, symbol, or almost any other key) so replica placement tends to be more explicit.

Map reduce is the typical way to build analytical applications for these data stores either using hadoop (Cassandra) or internal map reduce (MongoDB has a java script MR API). This is typically not the ideal paradigm for analyzing time series data.

Traditional column oriented stores with integrated time series analytics tools or custom built platforms will continue to be the preferred way to store and process time series data. I don't think that there will (or should) be much convergence between the databases built for running very high scale websites and the databases built to store and analyze time series information.

All that being said we use mongodb extensively for metadata storage and as a sort of very large cache. But we don't use it to store or analyze financial data.

share|improve this answer
1  
"Financial data tends to be small relative to large web companies data"... yeah tell that to the 60 nasdaq TAQ dvds sitting on my desk right now (2010 data only). All compressed about 95%.Tick data and even 1 minute data on a large number of firms can be quite data expensive. Also if you are doing various calculations on various periods, the standardized columns of mysql might be overbearing, when you can simply add any fields you need to a NoSql document (Mongodb specifically). – NoviceCoding Jan 31 '12 at 3:37
1  
ITCH, and OpenBook ULTRA, PITCH, etc... would not be considered big by web company standards. OPRA is a big feed. 60 DVDs is just under 300 GB compressed which is tiny. There are plenty of reasons to use NoSQL for the reason you mention which is no schema. However it is unlikely that we will see (or should see) large scale (thousands of machines in an instance) NoSql deployments in finance because the data simply isn't that big. – Steve Jan 31 '12 at 17:50
But would you say thats because of the fact that support/employees for NoSql are small? Or is it a NoSql infrastructural issue? For example mongodb has one of the easiest (from what I've read) sharding and replication mechanisms which make it very easy useful for multi machine expansion (one of the most mentioned resions to switch away from SQL to MongoDb). But I am saying this with only experience of starting a project I am working on in mongo (college student, no legit experience). – NoviceCoding Jan 31 '12 at 18:50
And in terms of speed, although many-many relations are not there, they say that indexed mongo docs are blazing fast (still not sure whether 'blazing fast' is 'quant fast' :) – NoviceCoding Jan 31 '12 at 18:52
2  
What I am saying is that many of the problems that the NoSQL data bases from the web world (Mongo, BigTable, Dynamo, their clones, etc...) are trying to solve are not necessarily the same as you would find in the financial world. There are another set of stores mentioned in Dirk's answer that are probably a better fit for quant problems. Multi machine expansion is not free and is easier in a NoSQL database because most make less strict guarantees. Mongo is very fast at key lookup. It does not have the type of time series operators that it would need to be truly useful. – Steve Jan 31 '12 at 19:40
show 1 more comment
up vote 3 down vote

In the finance world, most data (especially given this forum is for Quant monkeys) is transactional and subject to regulatory reporting.

NoSQL isn't generally transactional and given the shape of, say, risk data, there is no overpowering reason to throw away ACID and RDBMS.

There are more mundane reasons: there are 1000s of IT guys who know RDBMS. When your NoSQL folk leave, where is your continuity and support?

I'm biased here: I'm an RDBMS database specialist who manages risk/pricing/trading systems

share|improve this answer
2  
what about the times when you simply want to store and perform analysis on data. i'd argue that in this case going to RDBMS is slower. by the way i am not biased towards one or another. – user697697 Jul 1 '11 at 19:28
In the finance world, most quants use column-oriented storage. This is particularly true for time-series data, like tick history. – chrisaycock♦ Jul 2 '11 at 3:38
@chrisaycock: based on my experience of Derivatives IT, they're all on conventional RDBMS. Or Excel. – user1041 Jul 2 '11 at 9:09
based on my experience, the difference seems to be within people who make money with data, who uses advanced systems, and people who use client for data, who use good'ol technology.. – nicolas Jul 9 '11 at 17:31
"In the finance world, most data (especially given this forum is for Quant monkeys) is transactional and subject to regulatory reporting." NoSQL also allows for transactional reporting. For example, in Cassandra, simply force the replication factor to the same number as the number of nodes in the cluster and the consistency iis the same as SQL Server, Oracle, or MySQL. – Thomas Atwood Dec 10 '12 at 20:39
up vote 3 down vote

mongodb seems to be a good opensource solution to store historical tick-by-tick. I am using it here and it seems to be very simple to use.

share|improve this answer
How about the analytics side of things? Any successes or gotchas you can report? – chrisaycock♦ Dec 13 '11 at 17:04
what can I say ! its well developed using boost and a modern c++ approach and yes ! It does what they said on they site. it is scalable and high-performance. even the transport protocol is fast and it is JSON based so the maintenance of the database is quite simple. And I am not doing merchandise but the guys did a good work. And even better you can use it to implement the publish–subscribe pattern +.+ – Clebson Derivan Feb 27 at 21:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.