Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
50

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as people downvoted my answer. I certainly hope people are more open minded in the quant forum. Note - It seems that this suggestion has created some strong ...


48

Column-oriented storage is faster for reading because of the cache efficiency. Looking at your sample query: select price, time from data where symbol = `AAPL Here I'm concerned with three columns: price, time, and symbol. If all ticks were stored by row, the database would have to read through all rows just to search for the symbols. It would look like ...


42

Consider the standard error, and in particular the distance between the upper and lower limits: \begin{equation} \Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha \end{equation} Using the formula for standard error, we can solve for sample size: \begin{equation} n = \left(\frac{2 \cdot s \cdot \alpha}{\Delta}\...


39

My deal is HFT so what I care about is read/load data from file or DB quickly in memory perform very efficient data-munging operations (group,transform) visualize easily the data I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort. About 1. and 2. I am amazed reading ...


38

This isn't really an answer, but it's too long to add as a comment. I've always had a real problem with the correlation/covariance of price. To me, it means nothing. I realize that it gets used (abused) in many contexts, but I just don't get anything out of it (over time, price has to generally go up, go down, or go sideways, so aren't all prices "...


27

I have long hungered for the ultimate, super-fast, super-scaleable data storage solution. I have used relational databases, kdb, flatfiles, and binary files. In the end, I used binary files in my research language of choice. My advice is to KISS. The choice of storage is actually not that critical (unless maybe you're working with options tick data). ...


23

I would consider a motion chart that plots the eigenvalues of the covariance matrix over time. For a static view you can create a table: rows represent dates, and columns represent eigenvectors. The entries of the table represent changes in the angle of the eigenvector from the previous row. This will show how stable your covariance structure is. You can ...


23

Instead of wild guesses about R's/python's future in the community, here some facts: The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis. You can easily run this query on ...


21

One of my favorites is a generalization of correlation: Distance Correlation (dCor) There are several reasons for that: It generalizes classical (i.e. linear) correlation in the sense that linearity is a special case. It gives identical readings for linear dependence. There are analogs for variance, covariance and standard deviation, so these identities ...


21

This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story. I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $\mathbb{P}$ area, with some excursions into $\mathbb{Q}$). I tried Python but it didn't work for me. I think the main reasons I ...


21

I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis. While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding ...


20

Variance ratio tests have been used numerous times to show that financial asset prices do not follow a random walk. You can for example look at -Lo and MacKinlay : Stock market prices do not follow a random walk : http://press.princeton.edu/books/lo/chapt2.pdf (US Stocks) -Hoque, Kim, Pyun: A comparison of variance ratio tests of random walk: A ...


16

Personally I make a distinction between two conflicting goals: (1) storing data incoming in real-time for immediate processing and (2) storing the gathered data for "offline" purposes. Such approach makes things a lot easier if we're talking about a home-grown solution. (1) must be as fast as possible but not necessarily scalable beyond a few dozen millions ...


16

You can use changepoint analysis to identify regime change. You can also look at large angle differences in the eigenvectors between your most up-to-date/recent covariance matrix and the covariance matrix from the prior window. Another way to identify regime change is using a factor model. If the returns on a particular set of factors is X standard ...


16

There are many different methods for this. Most people rely on a unit root test. Rmetrics has collected the most common unit root tests into the fUnitRoots package, which primarily provides a wrapper for Bernhard Pfaff's urca package. These include: Augmented Dickey–Fuller (ADF) test Elliott–Rothenberg–Stock test KPSS unit root test Phillips–Perron ...


15

As a good starting point read this recent paper by Jing Chen: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1734526 For a special use of the entropy concept for forecasting the '87-crash read this paper: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=959547 (Although I tried to contact the authors to get the data to reproduce their findings, which ...


15

Correlation is much more widely used concept and it has much more "informal" meanings. If we have only two random variables $X$ and $Y$ then correlation is simply a measure of linear dependence between the two variables: $$corr(X,Y)=\frac{cov(X,Y)}{\sqrt{var(X)var(Y)}}=\frac{EXY-EX\cdot EY}{\sqrt{var(X)var(Y)}}$$ If correlation is -1 or 1 then the two ...


14

You can forecast stock prices thru time-series models, cross-sectional, or panel models. There is considerable variation within these categories. In time-series models you would use an auto-regressive model such as an AR(1) where the independent variable is the dependent variable lagged by one period. Naturally, an AR(2) would consist of 2 lags and so on. ...


14

I don't know how to select ARMA lag length when doing ARMA-GARCH. Perhaps someone can edit it into this answer. For the univariate case you want rugarch package. If you're doing multivariate stuff you want rmgarch. The reason these are better than other packages is threefold; (i) Support for exogenous variables which I haven't seen in any other package, (ii)...


14

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use case. If someone says they get a "x ms read", "y inserts per second", "k times speedup", "store n TB data" or "have m years of experience" and use that to ...


13

From Quantitative Trading by Ernie Chan : "Correlation between two price series actually refers to the correlations of their returns over some time horizon (for concreteness, let's say a day). If two stocks are positively correlated, there is a good chance that their prices will move in the same direction most days. However, having a positive correlation ...


13

Actually, co-skewness is represented by a rank 3 tensor, rather than a matrix. I'm going to reproduce the formulation from Bhandari and Das, Options on portfolios with higher-order moments, but I'll add and omit some details. The co-skewness tensor is $$ S_{ijk} = E \left[ r_i \times r_j \times r_k \right] = \frac{1}{T} \sum_{t=1}^T r_i(t) \times r_j(t) \...


13

You could try Arctic. Other open source column-oriented databases that you may not have considered include LucidDB and C-Store.


12

I think there are a lot of different ways to specify this problem. For simplicity, consider independent Garch processes $$ r_{1,t} \sim N\left(0,\sigma_{1,t}^{2}\right) $$ $$ \sigma_{1,t}^{2} = \beta_{1,1}+\beta_{1,2}\varepsilon_{1,t-1}^{2}+\beta_{1,3}\sigma_{1,t-1}^{2} $$ and $$ r_{2,t} \sim N\left(0,\sigma_{2,t}^{2}\right) $$ $$ \sigma_{2,t}^{2} = \beta_{...


12

Basically, prices usually have a unit root, while returns can be assumed to be stationary. This is also called order of integration, a unit root means integrated of order 1, I(1), while stationary is order 0, I(0). Time series that are stationary have a lot of convenient properties for analysis. When a time series is non-stationary, then that means the ...


12

For data analysis, particularly for large data analysis project, pretty much most of the top quant hedge funds and a lot of the banks are using Python (over R) for a couple of reasons but many still have bits and pieces of R for specific packages or functions (I work at a bank and interface with quite a few quant hedge funds on data analysis): Earlier ...


11

I have been using FastBit for a while now and find it to be quite performant. It's very non-intrusive to your existing binary storage format provided your data is stored in a columnar manner. I have briefly tested Tokyo/KyotoCabinet and didnt find it suitable for my (persistent storage) requirements.


11

I really wouldn't implement time series on my own unless I had a good reason to. AQR uses pandas, almost everyone in R using zoo or xts. I never like multiple parallel arrays, if it breaks everything is broken, plus it gets uglier as you increment data. If you are doing something in C++, why not have an array of structs for each object where you have ...


11

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they store data along one of the dimensions (which you must choose, usually time) contiguously on disk, and thus reads are extremely fast. The alternative, when ...


10

If you are serious about performance and flexibility, you have to take a look at data.table package in R. Here is the crantastic review. It is lighting fast! I think this is the best package addressing performance and memory issues.


Only top voted, non community-wiki answers of a minimum length are eligible