How do you evaluate a covariance forecast?

Question

Suppose you have two sources of covariance forecasts on a fixed set of $n$ assets, method A and method B (you can think of them as black box forecasts, from two vendors, say), which are known to be based on data available at a given point in time. Suppose you also observe the returns on those $n$ assets for a following period (a year's worth, say). What metrics would you use to evaluate the quality of these two covariance forecasts? What statistical tests?

For background, the use of the covariances would be in a vanilla mean-variance optimization framework, but one can assume little is known about the source of alpha.

edit: forecasting a covariance matrix is a bit different, I think, than other forecasting tasks. There are some applications where getting a good forecast of the eigenvectors of the covariance would be helpful, but the eigenvalues are not as important. (I am thinking of the case where one's portfolio is $\Sigma^{-1}\mu$, rescaled, where $\Sigma$ is the forecast covariance, and $\mu$ is the forecast returns.) In that case, the metric for forecasting quality should be invariant with respect to scale of the forecast. For some cases, it seems like forecasting the first eigenvector is more important (using it like beta), etc. This is why I was looking for methods specifically for covariance forecasting for use in quant finance.

Answer 1

You are correct: evaluating volatility forecasts is quite different from evaluating forecasts in general, and it is a very active area of research.

Methods can be classified in several ways. One criterion is to consider evaluation methods for single forecasts (e.g., for the time series of returns of a specific portfolio) vs multiple simultaneous forecasts (e.g., for an investable universe). Another criterion is to separate direct evaluation methodsfrom indirect evaluation methods (more on this later).

Focusing on single-asset methods: historically the most commonly used approach by practitioners, and the one advocated by Barra is the "bias" statistics. If you have a forecast return process $r_t$ and a forecast $h_t$, then under the null hypothesis that the forecast is correct, $r_t/h_t$ has unit variance. The Bias statistics is defined as $T^{-1} \sum_{t=1}^T (r_t/h_t)^2$, which is asymptotically normally distributed with unit mean and st.dev. $1/\sqrt{T}$, which can be used for hypothesis testing.

An alternative is Mincer-Zarnowitz regression, in which one runs a regression between realized variance (say, 20-trading day estimate between $t$ and $t+20$) and the forecast:

$$\hat\sigma^2_t =\alpha +\beta h^2_t + \epsilon_t$$

Under the null one tests the joint hypothesis $\alpha=0, \beta=1$. Patton and Sheppard also recommend the regression, which yields a more powerful test:

$$(\hat\sigma_t/h_t)^2 =\alpha/h^2_t +\beta + \epsilon_t$$

Both these tests can be (non-rigorously) extended to multiple forecasts by simulating random portfolios and generating statistics for each portfolio, or by assuming an identical relationship between forecasts and realizations across asset pairs: $$vech(\Sigma_t) = \alpha + \beta vech ( H_t) + \epsilon_t$$ in which $vech$ is the "stacking" operator on a matrix (in this case, the forecast and realized variance matrices).

As for indirect tests, a popular approach is the minimum variance portfolio for risk model comparison. One finds the minimum variance portfolio under a unit budget constraint using two or more asset covariance matrices. One can prove that the true covariance matrix would result in the portfolio with the lowest realized variance. In other words, better models hedge better. The advantage of this approach is that it does take into account the quality of the forecast of $\Sigma^{-1}$, which is used in actual optimization problems; and it doesn't require providing alphas, so that the test is not a joint test of risk model and manager skill.

Answer 2

You probably want to take it back to how one evaluates forecast models in general: using some metrics over one- or many-step forecasts, see e.g. here for a Wikipedia discussion. But instead of forecasting first moments, it would now be second moments.

This can still use (root) mean squared error, or mean absolute percentage error, or related measures; see e.g. this paper by Rob Hyndman on comparisons of methods.

Answer 3

I think a good approach is to compare your two covariance matrices on a set of random portfolios (see for instance http://www.portfolioprobe.com/about/applications-of-random-portfolios/assess-risk-models/).

What you want is a high correlation (across the portfolios) between the predicted and realized portfolio volatility. We're never going to estimate the level of volatility especially well. But if you get the right ranking across portfolios, then that is as much as you can ask.

It would be best to generate random portfolios that look like the ones you will actually have, but even naively generated portfolios may be good enough.

Answer 4

In mean-variance portfolio work, the elements of the covariance matrices are highly volatile and infused with error, so how to obtain forecasts that are usable ? A simple idea is to use a Stein-equal covar shrinkage estimator which, in practice, is easy to calculate and produces superior portfolios when evaluated on out-of-sample data ( see Continuous Time Mean Variance Portfolios, Zhou, 2000) . So to evaluate a proposed covar matrix,:

1) calculate the equal covar matrix, covar.eq, where covar.eq[i,i] = covar[i,i]; and covar.eq[i,j] = mean(covar[i,j]); i not = j.

2) del = Sum(i,j)[ Abs(covar.eq[i,j]-covar[i,j] )], or if you prefer delsq = sum(i,j)[ (covar.eq[i,j]-covar[i,j])^2 ]

and pick the covar with the smallest del or delsq. The selected covar matrix will be closest to the stein-shrinkage matrix which (see above ) produces superior portfolio's.

Paul H. Lasky B & P Investments

Answer 5

I like the ideas above but I am wondering why the obvious approach, likelihood, hasn't been mentioned in these answers?

I benchmark dozens of approaches and also the portfolio implications, and I'm finding that likelihood isn't a bad predictor of utility for the latter.

I mean the posterior gaussian likelihood that assumes zero mean (or some small mean perhaps if you really car). There are two terms to the likelihood aside from a normalizing constant

1. Log determinant (you can use fast_logdet from sklearn)
2. Multiplication of precision by data scatter

Full code and derivation.