Portfolio Optimisation/Covariance Estimation on a large scale

Question

When using Markowitz Portfolio Theory, e.g. for finding an optimal portfolio composition, one needs to have estimates of the returns, but most importantly of the covariance matrix. If our universe of assets/securities was, say, larger than 10,000 names (or just a very large number), how would one effectively come up with a useable estimate for the covariance matrix?

Clearly, we can use historically-observed covariance/correlations but given that we have such a large number of names, how accurate would these estimates be? Is it possible to reduce dimensionality by using some sort of PCA approach?

I am not interested in a perfect solution, but rather in ideas and potential techniques that I could read up on w.r.t. portfolio optimisation and parameter estimation on a large scale.

Answer 1

Broadly speaking, as you probably already know, there are 2 approaches to estimating large covariance matrices:

1) Shrinkage Methods like Ledoit-Wolf that try to reduce the noise in a large matrix (N by N) that has been estimated using the conventional method.

2) Factor Models of Covariance as described in for example Connor Korajczik 2007 that assume that only a small number of factors matter. This is pretty much what you have in mind when you mention "some sort of a PCA approach". A decent paper on estimating such models is Extracting factors from heteroskedastic asset returns, by Christopher S. Jones

On the practical side there is a big project at Blackrock directed by Andrew Ang that is developing and updating a huge factor-based risk model for a large number of markets that is the biggest covariance estimation that I know of. Bloomberg is also developing a large risk model of this kind.

There are a lot of practical issues that you have to deal with if you do the estimation yourself. One is that not all assets have been in existence for the same length of time (Stambaugh wrote about this in 'Analyzing Investments Whose Histories Differ in Length' in 1997). Another is that daily correlations between stocks trading on different exchanges are tricky because, for example the trading hours for New York and Tokyo are quite different. Also, because of these and other problems the covariance matrix you estimate often turns out to be non-positive definite, which causes problems when you run an optimizer. You have to check the matrix before you use it and fix it to be positive definite if necessary.