# Tag Info

96

I am going to recommend something that I have no doubt will get people completely up in arms and probably get people to attack me. It happened in the past and I lost many points on StackOverflow as people downvoted my answer. I certainly hope people are more open minded in the quant forum. Note - It seems that this suggestion has created some strong ...

47

My deal is HFT so what I care about is read/load data from file or DB quickly in memory perform very efficient data-munging operations (group,transform) visualize easily the data I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort. About 1. and 2. I am amazed reading ...

33

All of the answers above (unfortunately highly upvoted at this point) are missing the point. You shouldn't pick a DBMS or storage solution by general performance benchmarks, you should pick it by use case. If someone says they get a "x ms read", "y inserts per second", "k times speedup", "store n TB data" or "have m years of experience" and use that to ...

29

Instead of wild guesses about R's/python's future in the community, here some facts: The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis. You can easily run this query on ...

28

One of my favorites is a generalization of correlation: Distance Correlation (dCor) There are several reasons for that: It generalizes classical (i.e. linear) correlation in the sense that linearity is a special case. It gives identical readings for linear dependence. There are analogs for variance, covariance and standard deviation, so these identities ...

25

This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story. I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $\mathbb{P}$ area, with some excursions into $\mathbb{Q}$). I tried Python but it didn't work for me. I think the main reasons I ...

23

I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis. While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding ...

16

Basically, prices usually have a unit root, while returns can be assumed to be stationary. This is also called order of integration, a unit root means integrated of order 1, I(1), while stationary is order 0, I(0). Time series that are stationary have a lot of convenient properties for analysis. When a time series is non-stationary, then that means the ...

15

The standard answer is going to be that for time series, you want a column store database. These are optimized for range queries (ie: give me everything between two timestamps) because crucially, they store data along one of the dimensions (which you must choose, usually time) contiguously on disk, and thus reads are extremely fast. The alternative, when ...

15

Interesting debate and Not to wake sleeping dogs, the world has moved quite a bit in the 1.5 years, and the data space has exploded. I would like to recommend some new technologies and at the same time share a few of my experiences in this space. As @madilyn is trying to explain: It all depends on your use case. In my experience it's easy to know what you ...

14

You could try Arctic. Other open source column-oriented databases that you may not have considered include LucidDB and C-Store.

13

Let’s take a simple example to answer a broad but interesting question: Imagine that we have a daily return serie denoted $r_{t}$ ( which is assumed to be stationary) and let's take a little time to define main concepts : Mean Process (First moment process) The unconditional mean of $r_{t}$ denoted $u$ is just its expectation $E(r_{t})$. It is not time ...

13

For data analysis, particularly for large data analysis project, pretty much most of the top quant hedge funds and a lot of the banks are using Python (over R) for a couple of reasons but many still have bits and pieces of R for specific packages or functions (I work at a bank and interface with quite a few quant hedge funds on data analysis): Earlier ...

12

I will assume a white noise is a process $(\varepsilon_t)$ with zero mean, no autocorrelation and constant variance $\sigma^2 > 0$ while a random walk is a process $(x_t)$ defined by $$x_{t+1} = x_t + \varepsilon_{t+1}$$ where $\varepsilon$ is a white noise. 1) No since $Var(x_{t+1}) = Var(x_t) + Var(\varepsilon_{t+1})$ is stricly increasing while ...

12

There is a deeper issue. Frequentist distributions are not probability distributions because they are designed to be minimax distributions rather than actual distributions. This ignores all of the other problems and this also ignores risk-neutral versus any other measure of risk aversion. An even deeper issue is that these models presume that the ...

11

Define excess return $r^x_{it} = r_{it} - r^f_{t}$ as the return $i$ minus the risk free rate, and $f_{jt}$ similarly denotes the excess return of factor $j$ at time $t$. Let's say we have some factor model of returns where: $$r^x_{it} = \alpha_i + \sum_j \beta_{i,j} f_{jt} + \epsilon_{it}$$ F-test / GRS Test If we assume the error terms $\epsilon_{it}$ ...

10

There is a lot of ways to understand why stationarity allows to apply usual time series analysis. Here is one more. Very often, the theoretical justification of what you do in time series need to be able to identify the mean formula and the expectation: $$\frac{1}{N}\sum_{n=1}^N X_n \underset{N\rightarrow +\infty}{\longrightarrow} \mathbb{E} X,$$ where the ...

10

Not so fast! I think it is of the utmost importance to first examine whether the data points are real outliers, i.e. noise that is contaminating the data, or perhaps the most important pieces of the time series! For example when you look at US stock market data of the last 50 years and remove only the ten biggest moves because they are outliers you get a ...

10

The dollar bars certainly allow for a partial recovery of normality through a price sampling process subordinated to a volume, tick, dollar clock. It is well known that returns are assumed to be normally distributed but in reality they have a high kurtosis and fat tails, they are leptokurtic. Dr de Prado posits in several papers but mainly in a well known ...

9

The best paper is probably Relative Volume as a Doubly Stochastic Binomial Point Process - James Mcculloch. In this paper the volume is modelled via a Point Process, and theoretical laws are derived (with confident intervals, etc). And we put elements about this in Market Microstructure in Practice, Chap 2.1. Volume curves are analyzed, not only during the ...

9

Convenient rewriting Let $$X_t = c + \phi_1 X_{t-1} + \epsilon_t, \quad \vert \phi_1 \vert < 1 \tag{1}$$ denote a weakly stationary AR(1) process. Weak stationarity notably implies that $$\Bbb{E}[X_t] = \mu = \text{constant}$$ for all $t$. This property may be used (simply take the expectation on both sides of equation $(1)$), to find that the stationary ...

8

It appears that you are re-running the regression with each new data point. Instead, you should use an update/online formula (see an excellent answer by the famous Dr. Huber at stats.se). You can find an implementation in the R package biglm. If it doesn't have all the features you need (no windowing out of old data) you can at least adapt it and use it ...

8

I would suggest Time Series Analysis by James Douglas Hamilton

8

Perhaps overly simplistic and repeating the pt above, but when doing statistics, ideally we want to compare like with like. Returns can be comparable with each other. Prices on the other hand always depend on the previous price.

8

Mean reversion speed $\kappa$ is better interpreted with the concept of half-life, which can be calculated from $\text{HL} = \ln(2) / \kappa$. For example, if the mean reversion coefficient is $\kappa = 1.5$, then the half-life of the process is $\ln(2) / 1.5 = 0.46209812$ years, or about 6 months. Let's assume that the current interest rate is 1% and the ...

8

OpenTSDB is good for large-scale time series storage. metrilyx/opentsdb-pandas and wiktorski/opentsdb_pandas seems to provide the interface with pandas. OpenTSDB and HBase rough performance test | MoreDevs provides a benchmark, may not exactly match your requirements but you can try.

7

I work with time series intensively, and I am experienced in Java and scripting languages such as MATLAB and R. I strongly suggest that you should cook up your own implementations in Java, and stop hunting for and relying on any off-the-shelf implementations. They are not reliable. One should be able to write std, corr, cov, ma, etc., easily by hand. Coding ...

7

What you need is more mutual information rather than Shannon entropy. It is dedicated to capture the influence of one variable on another (you can think about it as a non linear version of Pearson correlations). They are closely related since the mutual information $I$ between two variables $X$ and $Y$ reads: $$I(X;Y) = H(X,Y) - H(X|Y) - H(Y|X)$$ where $H$ ...

7

In this context, unconditional variance refers to the stationary variance level predicted by your GARCH model. This quantity need not coincide with the sample variance of the data on which the latter model has been calibrated. That being said, in an effort to reduce the complexity of the GARCH parameters' estimation process (nasty non-linear optimisation ...

7

In their book "Counterparty Credit Risk, Collateral and Funding" D. Brigo, M. Morini and A. Pallavicini start with a dialogue between a Physics PhD graduate and an experienced practitioner of Quantitative Finance. The topic of P vs Q is presented in that dialogue in a manner meant to be understandable to a new comer. I would certainly recommend you to have ...

Only top voted, non community-wiki answers of a minimum length are eligible