# Tag Info

36

Column-oriented storage is faster for reading because of the cache efficiency. Looking at your sample query: select price, time from data where symbol = `AAPL Here I'm concerned with three columns: price, time, and symbol. If all ticks were stored by row, the database would have to read through all rows just to search for the symbols. It would look like ...

31

This one is quite easy: Think of a man walking his dog. He will go along and his dog will stroll along running back and forth. Man and dog are mathematically "cointegrated". As an investor you bet that the dog is coming back to his master or that the leash has only a certain length.

30

Consider the standard error, and in particular the distance between the upper and lower limits: $$\Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha$$ Using the formula for standard error, we can solve for sample size: n = \left(\frac{2 \cdot s \cdot \alpha}{\Delta}\...

28

The standard story (also told by @vonjd) is of "The Drunk and Her Dog". This is based on "A Drunk and Her Dog: An Illustration of Cointegration and Error Correction" (1994). The story is itself based on the standard illustration for a random walk known as the "drunkard's walk". The Dickey-Fuller test is used to check for a unit root. It can be used as ...

27

A stationary process is one where the mean and variance don't change over time. This is technically "second order stationarity" or "weak stationarity", but it is also commonly the meaning when seen in literature. In first order stationarity, the distribution of $(X_{t+1}, ..., X_{t+k})$ is the same as $(X_{1}, ..., X_{k})$ for all values of $(t, k)$. ...

26

This isn't really an answer, but it's too long to add as a comment. I've always had a real problem with the correlation/covariance of price. To me, it means nothing. I realize that it gets used (abused) in many contexts, but I just don't get anything out of it (over time, price has to generally go up, go down, or go sideways, so aren't all prices "...

24

Yes, there is such a rule and it is not too hard to grasp. Consider the 3-element correlation matrix $$\left(\begin{matrix} 1 & r & \rho \\ r & 1 & c \\ \rho & c & 1 \end{matrix}\right)$$ which must be positive semidefinite. In simpler terms, that means all its eigenvalues must be nonnegative. Assuming that $\rho$ and $r$...

23

I have long hungered for the ultimate, super-fast, super-scaleable data storage solution. I have used relational databases, kdb, flatfiles, and binary files. In the end, I used binary files in my research language of choice. My advice is to KISS. The choice of storage is actually not that critical (unless maybe you're working with options tick data). ...

20

I assume you're using returns (or log returns) instead of actual stock prices. In practice, you may also want to smooth the data by using a moving average. There are several correlation coefficients: Pearson's $r$ - most commonly used definition of correlation: $$r = \frac{\sigma_{x,y}}{\sigma_x \sigma_y}$$ Spearman's $\rho$...

20

I would consider a motion chart that plots the eigenvalues of the covariance matrix over time. For a static view you can create a table: rows represent dates, and columns represent eigenvectors. The entries of the table represent changes in the angle of the eigenvector from the previous row. This will show how stable your covariance structure is. You can ...

18

Representing time series (esp. tick data) using elaborate data structures may be not the best idea. You may want to try to use two arrays of the same length to store your time series. The first array stores values (e.g. price) and the seconds stores time. Note that the second series is monotonically increasing (or at least non-decreasing), i.e. it's sorted. ...

18

One of my favorites is a generalization of correlation: Distance Correlation (dCor) There are several reasons for that: It generalizes classical (i.e. linear) correlation in the sense that linearity is a special case. It gives identical readings for linear dependence. There are analogs for variance, covariance and standard deviation, so these identities ...

18

Variance ratio tests have been used numerous times to show that financial asset prices do not follow a random walk. You can for example look at -Lo and MacKinlay : Stock market prices do not follow a random walk : http://press.princeton.edu/books/lo/chapt2.pdf (US Stocks) -Hoque, Kim, Pyun: A comparison of variance ratio tests of random walk: A ...

18

My deal is HFT so what I care about is read/load data from file or DB quickly in memory perform very efficient data-munging operations (group,transform) visualize easily the data I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort. About 1. and 2. I am amazed reading ...

14

Two time series $X_1$ and $X_2$ are cointegrated if a linear combination $aX_1+bX_2$ is stationary i.e. it has constant mean, standard deviation and autocorrelation function for some $a$ and $b$. In other words, the two series never stray very far from one another. Cointegration might provide a more robust measure of the linkage between two financial ...

14

As a good starting point read this recent paper by Jing Chen: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1734526 For a special use of the entropy concept for forecasting the '87-crash read this paper: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=959547 (Although I tried to contact the authors to get the data to reproduce their findings, which ...

14

This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story. I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $\mathbb{P}$ area, with some excursions into $\mathbb{Q}$). I tried Python but it didn't work for me. I think the main reasons I ...

14

I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis. While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding ...

13

You can either reuse the last computed EMA, or fill-forward the previous period's sample data and recompute the EMA. I generally prefer the second option, which should cause a decay. Only go for the first option if your application won't change its logic based on missing data.

13

Personally I make a distinction between two conflicting goals: (1) storing data incoming in real-time for immediate processing and (2) storing the gathered data for "offline" purposes. Such approach makes things a lot easier if we're talking about a home-grown solution. (1) must be as fast as possible but not necessarily scalable beyond a few dozen millions ...

13

You can forecast stock prices thru time-series models, cross-sectional, or panel models. There is considerable variation within these categories. In time-series models you would use an auto-regressive model such as an AR(1) where the independent variable is the dependent variable lagged by one period. Naturally, an AR(2) would consist of 2 lags and so on. ...

13

You can use changepoint analysis to identify regime change. You can also look at large angle differences in the eigenvectors between your most up-to-date/recent covariance matrix and the covariance matrix from the prior window. Another way to identify regime change is using a factor model. If the returns on a particular set of factors is X standard ...

12

One of the reasons the ARCH family of models is used is that you only need price data to generate the model. These data exist back to the 1800s, so ARCH is great for looking at volatility over very long periods. I don't know that I'd say that the ARCH model has a lot of problems -- it solved the problem of not allowing volatility in time or in the level of ...

12

Correlation is much more widely used concept and it has much more "informal" meanings. If we have only two random variables $X$ and $Y$ then correlation is simply a measure of linear dependence between the two variables: $$corr(X,Y)=\frac{cov(X,Y)}{\sqrt{var(X)var(Y)}}=\frac{EXY-EX\cdot EY}{\sqrt{var(X)var(Y)}}$$ If correlation is -1 or 1 then the two ...

12

I don't know how to select ARMA lag length when doing ARMA-GARCH. Perhaps someone can edit it into this answer. For the univariate case you want rugarch package. If you're doing multivariate stuff you want rmgarch. The reason these are better than other packages is threefold; (i) Support for exogenous variables which I haven't seen in any other package, (...

12

Instead of wild guesses about R's/python's future in the community, here some facts: The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis. You can easily run this query on ...

11

Actually, co-skewness is represented by a rank 3 tensor, rather than a matrix. I'm going to reproduce the formulation from Bhandari and Das, Options on portfolios with higher-order moments, but I'll add and omit some details. The co-skewness tensor is  S_{ijk} = E \left[ r_i \times r_j \times r_k \right] = \frac{1}{T} \sum_{t=1}^T r_i(t) \times r_j(t) \...

11

There are many different methods for this. Most people rely on a unit root test. Rmetrics has collected the most common unit root tests into the fUnitRoots package, which primarily provides a wrapper for Bernhard Pfaff's urca package. These include: Augmented Dickey–Fuller (ADF) test Elliott–Rothenberg–Stock test KPSS unit root test Phillips–Perron ...

10

From Quantitative Trading by Ernie Chan : "Correlation between two price series actually refers to the correlations of their returns over some time horizon (for concreteness, let's say a day). If two stocks are positively correlated, there is a good chance that their prices will move in the same direction most days. However, having a positive correlation ...

10

Here's an interesting possibility: correlation network analysis + motion chart. Thanks to the hot research efforts in social network analysis (SNA), network analysis and graphics libraries such as R and Gephis are now easily accessible. I am well-versed in correlation analysis, and have a feeling that SNA can be effectively adapted for it. After all, the '...

Only top voted, non community-wiki answers of a minimum length are eligible