Is a more robust Covariance estimation possible?

Question

I'm working on a mean-variance optimization problem, but instead of financial securities I'm choosing a 'portfolio' of N athletes. It is a 1-period optimization problem over one generic statistic which I'll call performance here. I'm assuming athlete_performance is an N length random vector distributed as multivariate-normal:

athlete_performance $\sim MVN(\mu, \Sigma)$

Where $\mu$ is the 1xN vector of means (or expected performance)

and where $\Sigma$ is an NxN matrix with variance on the diagonal ( $\Sigma[i,i]$ = $Var(i)$ ) and covariance on the off-diagonal ( $\Sigma[i,j]$ = $Cov(i,j)$ ).

My question is about the options available for estimating the covariance (off-diagonal) part of the matrix.

My main concern is the predictiveness of my covariance matrix. If I were working with securities that had all been listed together for 10 years, then "sample covariance" might be predictive of future covariance, but in sports it's not that simple.

Imagine a quarterback and a wide receiver in American Football. How well their performance correlates is dependent on the quality of the pass defense they're playing against. Or in F1 racing, if driver A and B are both strong on straighter tracks, but only driver B is strong on tight cornered tracks, their performances will correlate much differently based on whether the track is straight or zigzag-ing.

I'm aware of "sample covariance" which in my case would look at the historical overlap between two athletes. I'm also aware of "shrunk covariance". I was wondering if there are more robust methods for calculating covariance that would be more predictive of future covariance, possibly using some sort of regression or MCMC.

Thank you for reading the question and for your time!

Answer 1

Quantile regression is considered a robust procedure but lacks the quality of being fully differentiable. There are also regularized regression models like ridge regression, lasso regression and elastic net regression that implicitly consider the covariance of the data like OLS, but additionally reduce volatility in estimates through the introduction of bias. These can account for correlation between time series like what you want, and have been shown to outperform OLS. Ridge regression specifically affects the diagonal of the covariance matrix to do this.

This bias-variance trade-off, typical in machine learning tasks, has a similar effect as covariance shrinkage, the best example being the Ledoit-Wolf covariance estimator that estimates the off-diagonal elements of the covariance matrix differently than the sample covariance estimator like what you want. Otherwise, you could look into eigenvalue techniques.

Answer 2

The Ledoit-Wolf estimate cited by @develarist can be quite good, but as you say you already knew about "shrinking". It takes the population of correlations observed as an effective Bayesian prior for any given correlation, so it sort of inherently assumes that all pairs are similar an some sense. It would not work well, say, with known block sets of highly correlated variables poisoning the sample set.

If you want something pairwise, say for variables $x$ and $y$, and therefore insensitive to population, consider the ideas of Gnanadesikan and Kettenring. Let's say you have a location (average) estimator $\mu(\cdot)$ and a scale (variability) estimator $\sigma(\cdot)$.

If $\sigma$ were standard deviation, then you could write the variance as

$$ \mathrm{Cov}(x,y) = \frac14\left( \sigma^2\left(\frac{x}{\sigma(x)}+\frac{y}{\sigma(y)}\right)- \sigma^2\left(\frac{x}{\sigma(x)}-\frac{y}{\sigma(y)}\right) \right) $$

(You don't actually need the $\mu()$.)

Thus, If you substitute some robust scale estimator $s(\cdot)$ for $\sigma(\cdot)$, you end up with robust pairwise covariance estimates.

If you need a positive semidefinite matrix of these, you will have to apply further postprocessing by orthogonal projection or the Higham algorithms.

For further information, see the rrcov R package documentation, or this stats.se answer: robust-covariance-and-ogk-outlier-detection.

Answer 3

As an addition to the already rich answers, I would suggest you to read the following paper by Marcos L. De Prado on the computation of Forward-Looking Correlation Matrices.

Estimation of Theory-Implied Correlation Matrices

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3484152

Answer 4

This is not a complete answer, more a different perspective to the answers already given. If you have some a-priori knowledge about the covariance structure and about the factors influencing it, you should try to reflect this in your statistical model. Three ideas:

• Divide your sample into subpopulations with identical factor values and estimate separately. In your example with the race tracks: Distinguish between performance correlation on straight and zig-zaggy tracks.
• Set up a low dimensional parametric model of your covariance matrix and regress the parameters of the matrix on the factors.
• Introduce additional variables to reflect the dependence directly. As an example in case $N=2$ instead of estimating $Cov(X,Y)$ define new "pure" performance variables $\tilde{X}$ and $\tilde{Y}$ which are uncorrelated and a common variable $Z$ such that $X=(1-\beta_X)\tilde{X} + \beta_XZ$ and $Y=(1-\beta_Y)\tilde{Y} + \beta_YZ$. Now estimate the two beta coefficients.

These are just some examples off my head what could be done. There are certainly many more (such as Bayesian methods).

Answer 5

I have actually considered the problem that you are working on, though configured somewhat differently.

There isn't going to be a universal answer to your question. See, in particular,

Holland, Paul W. Covariance Stabilizing Transformations. Ann. Statist. 1 (1973), no. 1, 84--92.

Nonetheless, there are answers, some already mentioned. I would argue that the purpose of the estimation should play a role in the answer. For example, if you worked for the Cincinnati Reds, then the only theoretically valid answer would be to use a Bayesian method because Frequentist methods are not coherent. That would bring up the question of loss functions and it is a generative method and not a sampling method. The concern would be how the data was generated and not the sampling properties of the estimator, which is really what you are asking about here.

On the other hand, if your concern were purely academic, there is a substantial advantage to Frequentist methods, despite the serious loss of fidelity to nature in exchange for accuracy through the loss in precision.

A downside of using such a method, though is that it would also be challenging to model competitive balance by considering the players that would be played against another set of players.

I think I should mention a bit on the difference of how robust methods versus less robust methods differs in the Frequentist and Bayesian perspective.

In the Frequentist case, robustness is often seen as a method that is good under weaker assumptions. The tradeoff is that predictive power falls in the case where a more efficient tool is available. Where the assumptions hold, ordinary least squares will provide the best prediction possible. When the assumptions become violated enough, a method such as Theil's regression becomes the most efficient estimator in terms of prediction.

On the Bayesian side, the tradeoffs between precision and accuracy are automatically made by the formula itself. Bayesian models are ex-post optimal models. If you build fragile Bayesian models, then you will get optimally constructed, fragile models. Robustness is created combinatorically by considering many alternative models, possibly under differing distributional assumptions.

On the Frequentist side, there is a tradeoff that you must make. Do you want robust estimators or estimators that are highly predictive?

The tradeoff on the Bayesian side is created by the posterior predictive distribution. It is the distribution of predictions that you would expect to see, having seen the data. There is an entire field in academia on scoring predictions. The tradeoff is in work performed. It takes a lot of computational work to produce very many predictions. Once that is done, it then takes an enormous amount of work to test the score of each prediction model.

A Frequentist solution may pop out in 2/10ths of a second and require trivial coding. A Bayesian solution may run for days before you get a solution and may require days of coding work if your model is not trivial.

One last note, let us assume $\Sigma_t\ne\Sigma_{t+1},\forall{t}$. In that case, your idea that a covariance estimator predicts a future covariance estimator doesn't really mean much because the estimators are estimating the current covariance matrix unless you build a regression model to predict the future time series.

Answer 6

For each of the examples provided in the question, if you had some view as to how X changes Y, you would update your $\mu$, not your $\Sigma$. Your covariance matrix by definition is meant to measure RANDOM noise between two variates.

Your sports examples are just as applicable to stocks. In the tech industry, company A manufactures parts overseas, but company B manufactures domestically. If the US is in a trade war with China, company A profits will covary much differently than if not.

With that said, if you're looking for a distribution of possible covariance (you mention MCMC), try Gibbs sampling using a Wishart prior for your covariance matrix. Practically, I'd suggest JAGS as a language to perform your sampling. You could also incorporate a linear regression inside your Gibbs algorithm and separate the samples into buckets with different conditions as @g g suggests.