39
My deal is HFT so what I care about is
read/load data from file or DB quickly in memory
perform very efficient data-munging operations (group,transform)
visualize easily the data
I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort.
About 1. and 2. I am amazed reading ...
25
Why is the first principal component a proxy for the market portfolio, and what other proxies exist?
Yes, the weights of the first eigenvector of a covariance matrix represent the market factor and also the largest source of systematic risk (variation of returns).
Why PCA? Well, PCA simply identifies the eigenvector that maximally explains the variance of the system. It turns out that this is the "market factor" - i.e. the tendency of securities to rise ...
23
Instead of wild guesses about R's/python's future in the community, here some facts:
The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis.
You can easily run this query on ...
21
It seems that your question refers to the microstructure noise defined in papers about intraday volatility estimates.
Originally, it comes from the bid-ask bounce, i.e. the fact that even if the volatility is zero, you have buyers and sellers at this price and consequently you observe prices at Bid or Ask prices, and not at mid-price. Because of that, if ...
21
This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story.
I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $\mathbb{P}$ area, with some excursions into $\mathbb{Q}$). I tried Python but it didn't work for me. I think the main reasons I ...
21
I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis.
While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding ...
20
Variance ratio tests have been used numerous times to show that financial asset prices do not follow a random walk. You can for example look at
-Lo and MacKinlay : Stock market prices do not follow a random walk : http://press.princeton.edu/books/lo/chapt2.pdf (US Stocks)
-Hoque, Kim, Pyun: A comparison of variance ratio tests of random walk: A ...
20
I'm sure Simons, as a first-rate pure and applied mathematician, had sufficient understanding of statistics to detect market inefficiencies and anomalies. As far as I know, the development and practice of statistical arbitrage as well as derivatives pricing has never been the exclusive domain of "outstanding probability or statistics professors."
Along with ...
17
Nick Higham's specialty is algorithms to find the nearest correlation matrix. His older work involved increased performance (in order-of-convergence terms) of techniques that successively projected a nearly-positive-semi-definite matrix onto the positive semidefinite space.
Perhaps even more interesting, from the practitioner point of view, is his ...
17
The term has a different meaning to different people.
to econometricians, microstructure noise is a disturbance that makes high frequency estimates of some parameters (e.g. realized volatility) very unstable. Generally this strand of the literature professes agnosticism as to the its origin;
to market microstructure researchers, microstructure noise is a ...
17
Some cynical but functional definitions:
It's what you can't model if you're not using tick by tick data
It's what proper quant pricing theory doesn't know how to model yet
It's information (order book behavior) that reflects momentary fluctuations in the supply/demand of a given contract, rather than its underlying value (eg an arbitrage free price)
The ...
17
Just to be painfully clear, it only seems to make sense to consider the logarithm of returns, i.e. $X=\log (1+\frac r{100})$ for a simple return of $r\%$ in an arbitrary period because this is what sums when returns are temporally aggregated. A basic property of cumulants is that cumulants of all orders are additive under convolution, for which a proof can ...
15
Correlation is much more widely used concept and it has much more "informal" meanings. If we have only two random variables $X$ and $Y$ then correlation is simply a measure of linear dependence between the two variables:
$$corr(X,Y)=\frac{cov(X,Y)}{\sqrt{var(X)var(Y)}}=\frac{EXY-EX\cdot EY}{\sqrt{var(X)var(Y)}}$$
If correlation is -1 or 1 then the two ...
15
From remote memory,
The first question is Yes/No question. Is there any stationary, i.e. I(0), time series for different levels of combination r? This question is answered by your first table.
For example, if [r=2]'s test stat is say 7 while the critical value of 99% confidence is 6.6 like your example, then I have over 99% confidence to say that all ...
15
The clearest and most intuitive article I have seen so far is
Kritzman et al., Regime Shifts: Implications for Dynamic Strategies in FAJ (May / June 2012)
It not only shows how you can use HMM for financial modelling but it also goes through the actual estimation algorithm (Baum-Welch) step-by-step and even gives full MATLAB-code.
From the abstract:
...
14
I don't know how to select ARMA lag length when doing ARMA-GARCH. Perhaps someone can edit it into this answer.
For the univariate case you want rugarch package. If you're doing multivariate stuff you want rmgarch. The reason these are better than other packages is threefold; (i) Support for exogenous variables which I haven't seen in any other package, (ii)...
13
Actually, co-skewness is represented by a rank 3 tensor, rather than a matrix.
I'm going to reproduce the formulation from Bhandari and Das, Options on portfolios with higher-order moments, but I'll add and omit some details.
The co-skewness tensor is
$$
S_{ijk} = E \left[ r_i \times r_j \times r_k \right] = \frac{1}{T} \sum_{t=1}^T r_i(t) \times r_j(t) \...
13
There are rigorous econometric definitions, as has already been eluded to by others. For practical purposes, microstructure noise is a component of a price process that exhibits mean reversion on some (possibly time-varying) frequency.
This reversion is particularly attractive to liquidity provisioners, who seek to profit from this noise component (along ...
13
You are correct that you can compute Sharpe ratios on portfolios with any return distribution. The issue is comparing Sharpe ratio's of non-normally distributed portfolios (which in reality is almost any portfolio). To take an extreme example. Consider two portfolios, with returns in excess of benchmark.
50% chance of 10% return, 50% chance of a 20% ...
12
There are a number of different tests that are generally used to compare samples to different distributions, such as Jarque-Bera, Anderson-Darling, and Kolmogorov–Smirnov (see this related question).
In your case, with just the standard deviation and mean, there isn't a whole lot to say. You need to assume a distribution (e.g. normal). You would be able ...
12
If the means are similar, then K-means will not do a great job. I would generate new features, perhaps based on higher moments of the distribution or some other properties (auto-correlation, summary of spectral density, etc.).
Using this new set of features, If you see separation of two curves when you plot draws in feature space then k-means would be an ...
12
In addition to John's answer and just to make things clear:
The arithmetic mean is given by
$$\mu_a = \frac{1}{n} \sum_{i=1}^n x_i$$
The geometric mean is given by
$$\mu_g = \sqrt[n]{\prod_{i=1}^n (1+x_i)} -1$$
And we have that
$$\mu_g \leq \mu_a$$
So not only would the geometric sharp ratio be taking into account the "actual" return of the ...
12
A simple google search should get your started:
I like this one the best because it compares different packages:
http://stat-www.berkeley.edu/~brill/Stat248/kalmanfiltering.pdf
and here couple more:
http://www.r-bloggers.com/the-kalman-filter-for-financial-time-series/
http://cran.r-project.org/web/packages/dlm/index.html
http://cran.r-project.org/web/...
12
For data analysis, particularly for large data analysis project, pretty much most of the top quant hedge funds and a lot of the banks are using Python (over R) for a couple of reasons but many still have bits and pieces of R for specific packages or functions (I work at a bank and interface with quite a few quant hedge funds on data analysis):
Earlier ...
10
This is indeed an interesting question.
According to this website, a paper by Goldman Sachs [Tierens and Anadu (2004)] proposes three alternative methods for estimating average stock correlations:
Calculate a full correlation matrix, weighting its elements in line
with the weight of the corresponding stocks in the portfolio/index,
and excluding ...
10
Jim Simons' initial intuitions about nonrandomness were probably driven by the very psychological/evolutionary predispositions to want to find the hidden meaning within noise that affect humanity in general. That Jim Simmons has been effective is a more a testament of his abilities and timing rather than his inclination to clinch that some patterns were not ...
9
Why don't you try it and report back?
Recall, though, that while a random walk is often a rather competitive forecast, realized data is understood to have weak dependence especially in higher moments.
Having worked a bit with DieHarder, I'd suspect it to reject a number of series. But the proof is in the pudding...
9
Before I try to answer your question we need to establish a difference between what one wants to analyse.
It is true that before modern time-series methodologies were developed, researches used "correlation" between prices as a means of analysis. However, since a Price (at a specific moment in time) is 1 value, it makes no sense to compare 2 prices with ...
9
Treat the estimate of standard deviation as a random variable. Then you can bootstap the sample estimate and generate t-statistics and associated confidence intervals for your statistics. I describe a generic boostrap process on this post.
9
I'm not sure it makes sense to think of one as more correct than another. However, they do have significant differences. It may help to distinguish between ex-post evaluation of a strategy and ex-ante prediction of what the strategy's performance will be.
For simplicity, let's assume the log returns of the strategy are approximately i.i.d. univariate ...
Only top voted, non community-wiki answers of a minimum length are eligible
