47
My deal is HFT so what I care about is
read/load data from file or DB quickly in memory
perform very efficient data-munging operations (group,transform)
visualize easily the data
I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort.
About 1. and 2. I am amazed reading ...
29
Instead of wild guesses about R's/python's future in the community, here some facts:
The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis.
You can easily run this query on ...
25
This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story.
I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $\mathbb{P}$ area, with some excursions into $\mathbb{Q}$). I tried Python but it didn't work for me. I think the main reasons I ...
23
I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis.
While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding ...
21
I'm sure Simons, as a first-rate pure and applied mathematician, had sufficient understanding of statistics to detect market inefficiencies and anomalies. As far as I know, the development and practice of statistical arbitrage as well as derivatives pricing has never been the exclusive domain of "outstanding probability or statistics professors."
Along with ...
16
You are correct that you can compute Sharpe ratios on portfolios with any return distribution. The issue is comparing Sharpe ratio's of non-normally distributed portfolios (which in reality is almost any portfolio). To take an extreme example. Consider two portfolios, with returns in excess of benchmark.
50% chance of 10% return, 50% chance of a 20% ...
15
The clearest and most intuitive article I have seen so far is
Kritzman et al., Regime Shifts: Implications for Dynamic Strategies in FAJ (May / June 2012)
It not only shows how you can use HMM for financial modelling but it also goes through the actual estimation algorithm (Baum-Welch) step-by-step and even gives full MATLAB-code.
From the abstract:
...
15
I don't know how to select ARMA lag length when doing ARMA-GARCH. Perhaps someone can edit it into this answer.
For the univariate case you want rugarch package. If you're doing multivariate stuff you want rmgarch. The reason these are better than other packages is threefold; (i) Support for exogenous variables which I haven't seen in any other package, (ii)...
13
A simple google search should get your started:
I like this one the best because it compares different packages:
http://stat-www.berkeley.edu/~brill/Stat248/kalmanfiltering.pdf
and here couple more:
http://www.r-bloggers.com/the-kalman-filter-for-financial-time-series/
http://cran.r-project.org/web/packages/dlm/index.html
http://cran.r-project.org/web/...
13
For data analysis, particularly for large data analysis project, pretty much most of the top quant hedge funds and a lot of the banks are using Python (over R) for a couple of reasons but many still have bits and pieces of R for specific packages or functions (I work at a bank and interface with quite a few quant hedge funds on data analysis):
Earlier ...
11
Have a look at this classic paper:
Honey, I Shrunk the Sample Covariance Matrix by O. Ledoit and M. Wolf
The abstract answers your question already:
The central message of this article is that no one should use the
sample covariance matrix for portfolio optimization. It is subject to
estimation error of the kind most likely to perturb a mean-...
11
This is indeed an interesting question.
According to this website, a paper by Goldman Sachs [Tierens and Anadu (2004)] proposes three alternative methods for estimating average stock correlations:
Calculate a full correlation matrix, weighting its elements in line
with the weight of the corresponding stocks in the portfolio/index,
and excluding ...
10
Jim Simons' initial intuitions about nonrandomness were probably driven by the very psychological/evolutionary predispositions to want to find the hidden meaning within noise that affect humanity in general. That Jim Simmons has been effective is a more a testament of his abilities and timing rather than his inclination to clinch that some patterns were not ...
9
I think this is a no-brainer. Only log-returns make sense. The average return can only be computed by averaging the sum of individual log returns. Taking the average of standard (relative) returns does not give you an average of the individual returns. Consider a simple case where the value of an investment alternates between 100 and 50 an odd number of ...
9
A great example of kalman filtering is in the Kyle Model. I have attached a presentation on the application of R to the kalman filter in the Kyle Model.
http://www.rinfinance.com/RinFinance2009/presentations/microstructure-tutorial.pdf
Basically in the Kyle Model, a market maker finds the likelihood an asset is ending up at a certain price given that a ...
9
We can obtain a closed-form expression for price correlation given (log) return correlation when the two stocks follow geometric Brownian motion:
$$S_1(t) = S_1(0)e^{(\mu_1- \frac{1}{2} \sigma_1^2)t}e^{\sigma_1Z_1(t)},\\ S_2(t) = S_2(0)e^{(\mu_2- \frac{1}{2} \sigma_2^2)t}e^{\sigma_2Z_2(t)},$$
where $\text{corr}(Z_1(t),Z_2(t)) = E[Z_1(t)Z_2(t)]=\rho t$. ...
8
I spoke to a quant trader from one of the leading quant proprietary trading firms about a year ago. Apparently, there was a point in time when they looked into applying machine-learning techniques to modelling problems. That would include things like neural networks, SVRs, graphical models and a bunch of other related stuff. These are essentially a kind of ...
8
Systematic Investor also did a two part series implementation in R which is also quite helpful as he details the pitfalls too.
Post One:
http://systematicinvestor.wordpress.com/2012/11/01/regime-detection/
Part Two:
http://systematicinvestor.wordpress.com/2012/11/15/regime-detection-pitfalls/
Updated version after the 'RHmm' library was taken down from ...
8
To clarify notation, you have an universe of $n=2000 \space$ stocks and two portfolio vectors $\mathbf{a},\mathbf{b}\in\mathbb{R}^{n}$ with $\left\|\mathbf{a}\right\|_{1}=\left\|\mathbf{b}\right\|_{1}=1$. Further, you have Estimators for the true Variance $\operatorname{Var}\left[\mathbf{a}\right]$ resp. $\operatorname{Var}\left[\mathbf{b}\right]$ and the ...
8
The estimation of a covariance matrix is unstable unless the number of historical observations $T$ is greater than the number of securities $N$ (5000 in your example). Consider that 10 years of data represents only 120 monthly observations and about 2500 daily observations.
Depending on the application, using data dating farther back than 10 years may be ...
8
I just want to add to vonjd's answer some info on the comparison of the 3 methods. This is too big for a comment so I'm posting as a separate answer but please upvote his answer, not mine.
Do the differences in methodologies matter in practice?
To gauge the practical importance of the biases in methods 2 and 3, we calculate the weighted stock correlation ...
7
I believe your conversation counter party did not mean to say that the theoretical nature of the applied statistical techniques are too advanced for a graduate math/stats/CS level student to master. I venture to assume what he/she meant was that the applied tool sets specifically to financial trading at that specific firm are/were too advanced for someone ...
7
The term in sample and out of sample are commonly used in any kind of optimization or fitting methods (MVO is just a particular case).
When you make the optimization, you compute optimal parameters (usually the weights of the optimal portfolio in asset allocation) over a given data sample, for example, the returns of the securities of the portfolio for the ...
7
If you consider $X_1$ a random variable which is normally distributed with mean $\mu$ and variance $\sigma^2$ them $S_1 = \exp(X_1)$ is log-normally distributed with mean $\exp(\mu + \sigma^2/2)$ and variance $(\exp(\sigma^2)-1)\exp(2\mu+\sigma^2)$. This follows from the definitions of the normal distribution and the log-normal distribution and deriving the ...
7
I would suggest Time Series Analysis by James Douglas Hamilton
7
Without seeing your trading desk's P&L it's impossible to say whether it is predictable or not. But here are a few thoughts -
There's no reason to think that it isn't predictable. In general, financial time series are hardest to predict when the represent the return stream of an investible asset. A trading desk's P&L isn't really investible, so ...
6
Its a simple expected value question:
Probability of throwing a 6 is 1/6
Probability of not throwing a 6 is 5/6
thus expected pay off per roll: 10 dollars * 1/6 + (-1 dollar) * 5/6 = 5/6 dollars
Edit: And several of your above assumptions are plain wrong:
"Probability tells me that every 6 throws I get one 6 and 5 different numbers." -> That is NOT what "...
6
To make things really clear, you have an original matrix $X$ of size $300 \times 10$ with all your returns.
Now what you do is that you choose the first $k=5$ eigenvectors (i.e. enough to get 80% of the variation given your data) and you form a vector $U$ of size $10 \times 5$. Each of the columns of $U$ represents a portfolio of the original dataset, and ...
6
If you wander about the theoretical result of fitting parameters, the book GARCH Models, Structure, Statistical Inference and Financial Applications of FRANCQ and ZAKOIAN provides a step-by-step explanation. I think that it is not a big problem to implement these steps to R.
6
This should walk you through what you are looking for:
https://www.quantstart.com/articles/Generalised-Autoregressive-Conditional-Heteroskedasticity-GARCH-p-q-Models-for-Time-Series-Analysis
https://www.quantstart.com/articles/ARIMA-GARCH-Trading-Strategy-on-the-SP500-Stock-Market-Index-Using-R
Only top voted, non community-wiki answers of a minimum length are eligible
