62

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as people downvoted my answer. I certainly hope people are more open minded in the quant forum. Note - It seems that this suggestion has created some strong ...


43

Consider the standard error, and in particular the distance between the upper and lower limits: \begin{equation} \Delta = (\bar{x} + SE \cdot \alpha) - (\bar{x} - SE \cdot \alpha) = 2 \cdot SE \cdot \alpha \end{equation} Using the formula for standard error, we can solve for sample size: \begin{equation} n = \left(\frac{2 \cdot s \cdot \alpha}{\Delta}\...


43

My deal is HFT so what I care about is read/load data from file or DB quickly in memory perform very efficient data-munging operations (group,transform) visualize easily the data I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort. About 1. and 2. I am amazed reading ...


26

Instead of wild guesses about R's/python's future in the community, here some facts: The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis. You can easily run this query on ...


22

One of my favorites is a generalization of correlation: Distance Correlation (dCor) There are several reasons for that: It generalizes classical (i.e. linear) correlation in the sense that linearity is a special case. It gives identical readings for linear dependence. There are analogs for variance, covariance and standard deviation, so these identities ...


22

This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story. I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $\mathbb{P}$ area, with some excursions into $\mathbb{Q}$). I tried Python but it didn't work for me. I think the main reasons I ...


21

I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis. While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding ...


17

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use case. If someone says they get a "x ms read", "y inserts per second", "k times speedup", "store n TB data" or "have m years of experience" and use that to ...


16

You can use changepoint analysis to identify regime change. You can also look at large angle differences in the eigenvectors between your most up-to-date/recent covariance matrix and the covariance matrix from the prior window. Another way to identify regime change is using a factor model. If the returns on a particular set of factors is X standard ...


16

There are many different methods for this. Most people rely on a unit root test. Rmetrics has collected the most common unit root tests into the fUnitRoots package, which primarily provides a wrapper for Bernhard Pfaff's urca package. These include: Augmented Dickey–Fuller (ADF) test Elliott–Rothenberg–Stock test KPSS unit root test Phillips–Perron ...


14

I don't know how to select ARMA lag length when doing ARMA-GARCH. Perhaps someone can edit it into this answer. For the univariate case you want rugarch package. If you're doing multivariate stuff you want rmgarch. The reason these are better than other packages is threefold; (i) Support for exogenous variables which I haven't seen in any other package, (ii)...


14

You could try Arctic. Other open source column-oriented databases that you may not have considered include LucidDB and C-Store.


13

For data analysis, particularly for large data analysis project, pretty much most of the top quant hedge funds and a lot of the banks are using Python (over R) for a couple of reasons but many still have bits and pieces of R for specific packages or functions (I work at a bank and interface with quite a few quant hedge funds on data analysis): Earlier ...


12

I think there are a lot of different ways to specify this problem. For simplicity, consider independent Garch processes $$ r_{1,t} \sim N\left(0,\sigma_{1,t}^{2}\right) $$ $$ \sigma_{1,t}^{2} = \beta_{1,1}+\beta_{1,2}\varepsilon_{1,t-1}^{2}+\beta_{1,3}\sigma_{1,t-1}^{2} $$ and $$ r_{2,t} \sim N\left(0,\sigma_{2,t}^{2}\right) $$ $$ \sigma_{2,t}^{2} = \beta_{...


12

Basically, prices usually have a unit root, while returns can be assumed to be stationary. This is also called order of integration, a unit root means integrated of order 1, I(1), while stationary is order 0, I(0). Time series that are stationary have a lot of convenient properties for analysis. When a time series is non-stationary, then that means the ...


12

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they store data along one of the dimensions (which you must choose, usually time) contiguously on disk, and thus reads are extremely fast. The alternative, when ...


11

I really wouldn't implement time series on my own unless I had a good reason to. AQR uses pandas, almost everyone in R using zoo or xts. I never like multiple parallel arrays, if it breaks everything is broken, plus it gets uglier as you increment data. If you are doing something in C++, why not have an array of structs for each object where you have ...


10

If you are serious about performance and flexibility, you have to take a look at data.table package in R. Here is the crantastic review. It is lighting fast! I think this is the best package addressing performance and memory issues.


10

You can use the (Adjusted) Dickey Fuller Test: http://en.wikipedia.org/wiki/Dickey%E2%80%93Fuller_test I'm pretty sure your software package has a library or routine you can use to do it.


10

Here is a structured list of your bullet points: covariance, correlation, PCA, factor analysis, Are similar. They are based on Gaussian assumptions (i.e. correlations means dependencies) and try to identify common factors (i.e. a variable in small dimension) explaining the observed relationships. co-integration is more specific in the sense that you ...


10

Not so fast! I think it is of the utmost importance to first examine whether the data points are real outliers, i.e. noise that is contaminating the data, or perhaps the most important pieces of the time series! For example when you look at US stock market data of the last 50 years and remove only the ten biggest moves because they are outliers you get a ...


9

Google for granger causality and its general version, transfer entropy, for a measure of whether a time series has a causal relationship with another (measured by calculating how much the conditional entropy of a time series decreases if we know another one, conditioned on everything else we know).


9

Have a look here: http://www.climatelogic.com/ The method is based on a sequential F-test, see also this paper: Rodionov, S.N., 2005b: Detecting regime shifts in the mean and variance: Methods and specific examples. In: Large-Scale Disturbances (Regime Shifts) and Recovery in Aquatic Ecosystems: Challenges for Management Toward Sustainability, V. Velikova ...


9

Two ways: Model the returns using an Ornstein-Uhlenbeck process You can control the variance of the residual noise in the process to your desired level of correlation. Conceptually you inject gaussian noise into the synthetic OU process to satisfy your requirement. For example, let's say you have time-series A which is what you are modelling. Time-series ...


9

GARCH will work if volume has memory with some decay. AR will work if volume has mean reversion properties. Both of these are empirical questions and depend on the market. You should also consider if there are seasonal (day-of-week, monthly, quarterly effects) in which case you would want to add dummy variables. MA models will work well if volume behaves ...


9

It only indicates that the null hypothesis of uncorrelated increments is violated. For the sake of simplicity, assume a time series is stationary. Then a sufficient statistic for arbitrary variance ratios is its covariance function. In general, a given deviation from the null can originate from different covariance functions, which in turn, entails that ...


9

Let’s take a simple example to answer a broad but interesting question: Imagine that we have a daily return serie denoted $r_{t}$ ( which is assumed to be stationary) and let's take a little time to define main concepts : Mean Process (First moment process) The unconditional mean of $r_{t}$ denoted $u$ is just its expectation $E(r_{t})$. It is not time ...


9

The best paper is probably Relative Volume as a Doubly Stochastic Binomial Point Process - James Mcculloch. In this paper the volume is modelled via a Point Process, and theoretical laws are derived (with confident intervals, etc). And we put elements about this in Market Microstructure in Practice, Chap 2.1. Volume curves are analyzed, not only during the ...


9

Define excess return $r^x_{it} = r_{it} - r^f_{t}$ as the return $i$ minus the risk free rate, and $f_{jt}$ similarly denotes the excess return of factor $j$ at time $t$. Let's say we have some factor model of returns where: $$ r^x_{it} = \alpha_i + \sum_j \beta_{i,j} f_{jt} + \epsilon_{it}$$ F-test / GRS Test If we assume the error terms $\epsilon_{it}$ ...


9

There is a deeper issue. Frequentist distributions are not probability distributions because they are designed to be minimax distributions rather than actual distributions. This ignores all of the other problems and this also ignores risk-neutral versus any other measure of risk aversion. An even deeper issue is that these models presume that the ...


Only top voted, non community-wiki answers of a minimum length are eligible