47
My deal is HFT so what I care about is
read/load data from file or DB quickly in memory
perform very efficient data-munging operations (group,transform)
visualize easily the data
I think is is pretty clear that 3. goes to R, graphics and ggplot2 and others allow you to plot anything from scratch with little effort.
About 1. and 2. I am amazed reading ...
30
Instead of wild guesses about R's/python's future in the community, here some facts:
The following query on StackExchange Data Explorer counts the number of questions that have <r> or <python> tags. If you scroll down on one of the three webpages provided below, you can see a graph with data on a monthly basis.
You can easily run this query on ...
25
This is interesting because I see another trend: Matlab is being replaced by R, but I guess this is another story.
I use R for my academic (I am also teaching this stuff) as well as my consulting work (I am mainly working in the $\mathbb{P}$ area, with some excursions into $\mathbb{Q}$). I tried Python but it didn't work for me. I think the main reasons I ...
23
I've used both R and Python with Pandas in a professional quantitative financial work to do both large and small scale projects. I would strongly recommend Python with Pandas over R for most new projects in the field especially in time series analysis.
While I don't dispute vonjd in that you will find more libraries in R with algorithms on the bleeding ...
22
I'm sure Simons, as a first-rate pure and applied mathematician, had sufficient understanding of statistics to detect market inefficiencies and anomalies. As far as I know, the development and practice of statistical arbitrage as well as derivatives pricing has never been the exclusive domain of "outstanding probability or statistics professors."
Along with ...
17
You are correct that you can compute Sharpe ratios on portfolios with any return distribution. The issue is comparing Sharpe ratio's of non-normally distributed portfolios (which in reality is almost any portfolio). To take an extreme example. Consider two portfolios, with returns in excess of benchmark.
50% chance of 10% return, 50% chance of a 20% ...
13
Have a look at this classic paper:
Honey, I Shrunk the Sample Covariance Matrix by O. Ledoit and M. Wolf
The abstract answers your question already:
The central message of this article is that no one should use the
sample covariance matrix for portfolio optimization. It is subject to
estimation error of the kind most likely to perturb a mean-...
13
For data analysis, particularly for large data analysis project, pretty much most of the top quant hedge funds and a lot of the banks are using Python (over R) for a couple of reasons but many still have bits and pieces of R for specific packages or functions (I work at a bank and interface with quite a few quant hedge funds on data analysis):
Earlier ...
11
This is indeed an interesting question.
According to this website, a paper by Goldman Sachs [Tierens and Anadu (2004)] proposes three alternative methods for estimating average stock correlations:
Calculate a full correlation matrix, weighting its elements in line
with the weight of the corresponding stocks in the portfolio/index,
and excluding ...
10
Jim Simons' initial intuitions about nonrandomness were probably driven by the very psychological/evolutionary predispositions to want to find the hidden meaning within noise that affect humanity in general. That Jim Simmons has been effective is a more a testament of his abilities and timing rather than his inclination to clinch that some patterns were not ...
9
I think this is a no-brainer. Only log-returns make sense. The average return can only be computed by averaging the sum of individual log returns. Taking the average of standard (relative) returns does not give you an average of the individual returns. Consider a simple case where the value of an investment alternates between 100 and 50 an odd number of ...
9
We can obtain a closed-form expression for price correlation given (log) return correlation when the two stocks follow geometric Brownian motion:
$$S_1(t) = S_1(0)e^{(\mu_1- \frac{1}{2} \sigma_1^2)t}e^{\sigma_1Z_1(t)},\\ S_2(t) = S_2(0)e^{(\mu_2- \frac{1}{2} \sigma_2^2)t}e^{\sigma_2Z_2(t)},$$
where $\text{corr}(Z_1(t),Z_2(t)) = E[Z_1(t)Z_2(t)]=\rho t$. ...
8
To clarify notation, you have an universe of $n=2000 \space$ stocks and two portfolio vectors $\mathbf{a},\mathbf{b}\in\mathbb{R}^{n}$ with $\left\|\mathbf{a}\right\|_{1}=\left\|\mathbf{b}\right\|_{1}=1$. Further, you have Estimators for the true Variance $\operatorname{Var}\left[\mathbf{a}\right]$ resp. $\operatorname{Var}\left[\mathbf{b}\right]$ and the ...
8
The estimation of a covariance matrix is unstable unless the number of historical observations $T$ is greater than the number of securities $N$ (5000 in your example). Consider that 10 years of data represents only 120 monthly observations and about 2500 daily observations.
Depending on the application, using data dating farther back than 10 years may be ...
8
I would suggest Time Series Analysis by James Douglas Hamilton
8
I just want to add to vonjd's answer some info on the comparison of the 3 methods. This is too big for a comment so I'm posting as a separate answer but please upvote his answer, not mine.
Do the differences in methodologies matter in practice?
To gauge the practical importance of the biases in methods 2 and 3, we calculate the weighted stock correlation ...
7
If you consider $X_1$ a random variable which is normally distributed with mean $\mu$ and variance $\sigma^2$ them $S_1 = \exp(X_1)$ is log-normally distributed with mean $\exp(\mu + \sigma^2/2)$ and variance $(\exp(\sigma^2)-1)\exp(2\mu+\sigma^2)$. This follows from the definitions of the normal distribution and the log-normal distribution and deriving the ...
7
Transaction costs - even for banks, funds etc, every trade has an associated cost, so if you would be buying a small number of shares, it's probably cheaper to carry the risk and not make those small trades.
The source data is imperfect, and contains noise. A lot of the smaller components are simply artefacts of that noise so it would be both an unnecessary ...
7
I think a good book to start in your case is:
Attilio Meucci: Risk and Asset Allocation
I once had a seminar held by Attilio that was based on the book and it blew my mind. The book is very intuitive yet rigorous.
7
Elements of Statistical Learning by Hastie, Tibshirani and Friedman is one of the most-cited books for your purpose. Although it does not have any direct applications to Finance, this is definitely a good book to have in your professional library and can be used as a reference for most topics.
If you want to use a book with more financial applications, I ...
7
Without seeing your trading desk's P&L it's impossible to say whether it is predictable or not. But here are a few thoughts -
There's no reason to think that it isn't predictable. In general, financial time series are hardest to predict when the represent the return stream of an investible asset. A trading desk's P&L isn't really investible, so ...
6
This should walk you through what you are looking for:
https://www.quantstart.com/articles/Generalised-Autoregressive-Conditional-Heteroskedasticity-GARCH-p-q-Models-for-Time-Series-Analysis
https://www.quantstart.com/articles/ARIMA-GARCH-Trading-Strategy-on-the-SP500-Stock-Market-Index-Using-R
6
If you wander about the theoretical result of fitting parameters, the book GARCH Models, Structure, Statistical Inference and Financial Applications of FRANCQ and ZAKOIAN provides a step-by-step explanation. I think that it is not a big problem to implement these steps to R.
6
Why do you have 16180 observations? Is this daily data over 64 years or higher frequency data? I am guessing so by the magnitude of the intercept. At any rate, your test power would be huge with this large sample size, meaning small relationships will be statistically significant.
What Cochrane said is contingent on data frequency. At a high frequency it ...
6
For the tasks listed, both Python and R perform very well. There are some packages in Python not in R and vice versa. My solution for this is to simply call R from Python. This allows for the best of both worlds.
It is also important to note I do not write any R code other than calling an R library from Python.
Calling Python from R does not work equally ...
6
Also in the high frequency / medium frequency field here.
I received a "mixed" consensus regarding the use of R and its prevalence in the field (specifically HFT). Speaking with someone who works in the equity option industry at a relatively small proprietary firm in San Francisco, I was told, "R is a legacy language".
However, speaking with someone who ...
6
Reading what I have, I can only offer a guess.
1: Let's say you're looking at 9 sectors compared to \$SPX on a daily chart. Foreach sector, compute relative closing price: 100 * Sector/\$SPX
2: It looks like the RS-Ratio is averaged over 14 periods. I say 14 because stockcharts.com shows RS-Ratio peaking after a lag (2-3wks), despite price peaking 2-3 ...
6
What you basically do here is a Principal Component Analysis (PCA). A good starting point in the financial sphere is
Managing Diversification by Attilio Meucci (2010)
Page 3:
"The most natural choice of uncorrelated risk sources is provided by the
principal component decomposition of the returns covariance [...] The eigenvectors define a set of N ...
6
I will disagree with RPL's answer - Simons is not particularly known as an applied mathematician, but he did work for some time at the Institute for Defense Analysis [IDA] (he was fired for insubordination), which is a signals intelligence shop. The game in signals intelligence is to find the signal in the noise, and that is exactly the game in quant finance ...
6
Yes. It comes from a core theorem of statics, Stein's Lemma. It shook the foundations of the field of statistics when it came out. It blew up an entire way of viewing mathematical statistics. Although it followed from critical work by Robbins in Bayesian estimation, Stein's work is really what is now remembered.
There are three primary schools of ...
Only top voted, non community-wiki answers of a minimum length are eligible