44
My deal is HFT so what I care about is
read/load data from file or DB quickly in memory
perform very efficient data-munging operations (group,transform)
visualize easily the data
I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort.
About 1. and 2. I am amazed reading ...
27
Instead of wild guesses about R's/python's future in the community, here some facts:
The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis.
You can easily run this query on ...
26
Why is the first principal component a proxy for the market portfolio, and what other proxies exist?
Yes, the weights of the first eigenvector of a covariance matrix represent the market factor and also the largest source of systematic risk (variation of returns).
Why PCA? Well, PCA simply identifies the eigenvector that maximally explains the variance of the system. It turns out that this is the "market factor" - i.e. the tendency of securities to rise ...
24
This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story.
I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $\mathbb{P}$ area, with some excursions into $\mathbb{Q}$). I tried Python but it didn't work for me. I think the main reasons I ...
22
I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis.
While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding ...
21
It seems that your question refers to the microstructure noise defined in papers about intraday volatility estimates.
Originally, it comes from the bid-ask bounce, i.e. the fact that even if the volatility is zero, you have buyers and sellers at this price and consequently you observe prices at Bid or Ask prices, and not at mid-price. Because of that, if ...
20
I'm sure Simons, as a first-rate pure and applied mathematician, had sufficient understanding of statistics to detect market inefficiencies and anomalies. As far as I know, the development and practice of statistical arbitrage as well as derivatives pricing has never been the exclusive domain of "outstanding probability or statistics professors."
Along with ...
17
The term has a different meaning to different people.
to econometricians, microstructure noise is a disturbance that makes high frequency estimates of some parameters (e.g. realized volatility) very unstable. Generally this strand of the literature professes agnosticism as to the its origin;
to market microstructure researchers, microstructure noise is a ...
17
Some cynical but functional definitions:
It's what you can't model if you're not using tick by tick data
It's what proper quant pricing theory doesn't know how to model yet
It's information (order book behavior) that reflects momentary fluctuations in the supply/demand of a given contract, rather than its underlying value (eg an arbitrage free price)
The ...
17
Just to be painfully clear, it only seems to make sense to consider the logarithm of returns, i.e. $X=\log (1+\frac r{100})$ for a simple return of $r\%$ in an arbitrary period because this is what sums when returns are temporally aggregated. A basic property of cumulants is that cumulants of all orders are additive under convolution, for which a proof can ...
15
The clearest and most intuitive article I have seen so far is
Kritzman et al., Regime Shifts: Implications for Dynamic Strategies in FAJ (May / June 2012)
It not only shows how you can use HMM for financial modelling but it also goes through the actual estimation algorithm (Baum-Welch) step-by-step and even gives full MATLAB-code.
From the abstract:
...
15
I don't know how to select ARMA lag length when doing ARMA-GARCH. Perhaps someone can edit it into this answer.
For the univariate case you want rugarch package. If you're doing multivariate stuff you want rmgarch. The reason these are better than other packages is threefold; (i) Support for exogenous variables which I haven't seen in any other package, (ii)...
13
There are rigorous econometric definitions, as has already been eluded to by others. For practical purposes, microstructure noise is a component of a price process that exhibits mean reversion on some (possibly time-varying) frequency.
This reversion is particularly attractive to liquidity provisioners, who seek to profit from this noise component (along ...
13
In addition to John's answer and just to make things clear:
The arithmetic mean is given by
$$\mu_a = \frac{1}{n} \sum_{i=1}^n x_i$$
The geometric mean is given by
$$\mu_g = \sqrt[n]{\prod_{i=1}^n (1+x_i)} -1$$
And we have that
$$\mu_g \leq \mu_a$$
So not only would the geometric sharp ratio be taking into account the "actual" return of the ...
13
A simple google search should get your started:
I like this one the best because it compares different packages:
http://stat-www.berkeley.edu/~brill/Stat248/kalmanfiltering.pdf
and here couple more:
http://www.r-bloggers.com/the-kalman-filter-for-financial-time-series/
http://cran.r-project.org/web/packages/dlm/index.html
http://cran.r-project.org/web/...
13
You are correct that you can compute Sharpe ratios on portfolios with any return distribution. The issue is comparing Sharpe ratio's of non-normally distributed portfolios (which in reality is almost any portfolio). To take an extreme example. Consider two portfolios, with returns in excess of benchmark.
50% chance of 10% return, 50% chance of a 20% ...
13
For data analysis, particularly for large data analysis project, pretty much most of the top quant hedge funds and a lot of the banks are using Python (over R) for a couple of reasons but many still have bits and pieces of R for specific packages or functions (I work at a bank and interface with quite a few quant hedge funds on data analysis):
Earlier ...
12
If the means are similar, then K-means will not do a great job. I would generate new features, perhaps based on higher moments of the distribution or some other properties (auto-correlation, summary of spectral density, etc.).
Using this new set of features, If you see separation of two curves when you plot draws in feature space then k-means would be an ...
11
I'm not sure it makes sense to think of one as more correct than another. However, they do have significant differences. It may help to distinguish between ex-post evaluation of a strategy and ex-ante prediction of what the strategy's performance will be.
For simplicity, let's assume the log returns of the strategy are approximately i.i.d. univariate ...
10
This is indeed an interesting question.
According to this website, a paper by Goldman Sachs [Tierens and Anadu (2004)] proposes three alternative methods for estimating average stock correlations:
Calculate a full correlation matrix, weighting its elements in line
with the weight of the corresponding stocks in the portfolio/index,
and excluding ...
10
Jim Simons' initial intuitions about nonrandomness were probably driven by the very psychological/evolutionary predispositions to want to find the hidden meaning within noise that affect humanity in general. That Jim Simmons has been effective is a more a testament of his abilities and timing rather than his inclination to clinch that some patterns were not ...
9
Treat the estimate of standard deviation as a random variable. Then you can bootstap the sample estimate and generate t-statistics and associated confidence intervals for your statistics. I describe a generic boostrap process on this post.
9
I think this is a no-brainer. Only log-returns make sense. The average return can only be computed by averaging the sum of individual log returns. Taking the average of standard (relative) returns does not give you an average of the individual returns. Consider a simple case where the value of an investment alternates between 100 and 50 an odd number of ...
9
A great example of kalman filtering is in the Kyle Model. I have attached a presentation on the application of R to the kalman filter in the Kyle Model.
http://www.rinfinance.com/RinFinance2009/presentations/microstructure-tutorial.pdf
Basically in the Kyle Model, a market maker finds the likelihood an asset is ending up at a certain price given that a ...
9
Have a look at this classic paper:
Honey, I Shrunk the Sample Covariance Matrix by O. Ledoit and M. Wolf
The abstract answers your question already:
The central message of this article is that no one should use the
sample covariance matrix for portfolio optimization. It is subject to
estimation error of the kind most likely to perturb a mean-...
8
"Treshold Garch" or T-Garch models are designed to capture this asymmetry. See this exposition by U. Chicago's Ruey Tsay who has a terrific text on time-series models in "Analysis of Financial Time Series".
You can use the structure of the T-Garch models to simulate data with this property.
There is a package called fGarch that creates APARCH models. A T-...
8
Attilio Meucci does some very interesting things with PCA. See e.g. his paper on managing diversification which makes heavy use of it (and explains it very intuitively along the way):
Managing Diversification by Attilio Meucci
8
Systematic Investor also did a two part series implementation in R which is also quite helpful as he details the pitfalls too.
Post One:
http://systematicinvestor.wordpress.com/2012/11/01/regime-detection/
Part Two:
http://systematicinvestor.wordpress.com/2012/11/15/regime-detection-pitfalls/
Updated version after the 'RHmm' library was taken down from ...
8
To clarify notation, you have an universe of $n=2000 \space$ stocks and two portfolio vectors $\mathbf{a},\mathbf{b}\in\mathbb{R}^{n}$ with $\left\|\mathbf{a}\right\|_{1}=\left\|\mathbf{b}\right\|_{1}=1$. Further, you have Estimators for the true Variance $\operatorname{Var}\left[\mathbf{a}\right]$ resp. $\operatorname{Var}\left[\mathbf{b}\right]$ and the ...
8
The estimation of a covariance matrix is unstable unless the number of historical observations $T$ is greater than the number of securities $N$ (5000 in your example). Consider that 10 years of data represents only 120 monthly observations and about 2500 daily observations.
Depending on the application, using data dating farther back than 10 years may be ...
Only top voted, non community-wiki answers of a minimum length are eligible
