Shrinkage of the Sample Covariance matrix, theory

Question

is there any theory behind the covariance matrix shrinkage paper, why it works?

I am talking about this stats exchange thread

Answer 1

Yes. It comes from a core theorem of statics, Stein's Lemma. It shook the foundations of the field of statistics when it came out. It blew up an entire way of viewing mathematical statistics. Although it followed from critical work by Robbins in Bayesian estimation, Stein's work is really what is now remembered.

There are three primary schools of statistics, the Frequentist, the Bayesian, and the Likelihoodists. The Likelihoodists are probably a minor school that bats in the big leagues sometimes, in terms of practical day-to-day usage. There are minor interpretations of probability like Rudolf Carnap's and there is still the minor school of statistics called Fiducial Statistics.

Each of these different schools actually solves different problems. It is a mistake to use them interchangeably but people do.

The 1930s through the 1950s were filled with foundational work in probability and statistics. One of the other key findings, which also sent shock waves through the field of statistics, was that all Bayesian statistics are admissible statistics and only those Frequentist statistics that are either identical at every sampling or match at the limit, are admissible.

There are a number of ways to talk about admissibility, but the simplest is to note its similarity to Pareto Optimality. A statistic is admissible if it cannot be stochastically dominated by another statistic.

In the simple world of 1950's statistics, work was done primarily on Frequentist statistics, but also on Likelihood-based statistics, such as Bayesian tools or the method of maximum likelihood. Lehman and Hodges showed there was a link between Bayesian statistics and sampling-based statistics like the maximum likelihood estimator (MLE) or the minimum variance unbiased estimator(MVUE).

A non-Bayesian estimator was admissible if it was a generalized Bayes rule. A statistic is Bayes, a Frequentist term for a particular type of utility function, if it does two things. Bayesian statistics minimize average loss, which is a scalar so it permits a total ordering of possible statistics. Frequentist statistics minimize the maximum risk, generally speaking, which is a function so does not guarantee a total ordering. You could impose a different criterion possibly. A generalized Bayes rule does both and maps to an implicit prior distribution such that the Bayesian answer and the Frequentist answer produce the same result.

That is a winded way of saying it doesn't matter which school of thought that you use, other than for interpretative purposes, in some elementary cases. If your only goal is a numerical estimation, then it doesn't matter how you solve your problem as long as you use a generalized Bayes rule.

In a world of punch card computing, that is the same as saying you can throw Bayesian statistics away. It is very difficult with modern computers to calculate a Bayesian answer. It is easy to calculate an MVUE usually.

All was going along swimmingly until two types of findings hit. The big one is Stein's Lemma. Stein's Lemma didn't just smack Frequentist statistics in the face, it uncovered issues in Bayesian methods too. Bayesian methods can have the integral in the denominator diverge when flat priors are used. That is the same as saying that sum of the probabilities does not add to one.

The second was that if real prior information did exist, then non-Bayesian estimators are inadmissible. That led to the creation of meta-analysis methodologies on the Frequentist side. There is a simple (or crazy hard depending on who you ask) method for a Bayesian researcher to incorporate the results of prior research directly into their calculations. Indeed, they are theoretically obligated to include prior findings in their statistical calculations, essentially as if they had observed the data in their own experiments.

One way you can view all Frequentist shrinkage estimators is as the limiting form of a Bayesian method that uses proper prior distributions. The difficulty is that there cannot be a canonical shrinkage estimator because there are an infinite number of proper prior distributions.

Assuming that you have never used a Bayesian method before, understand that it has three parts. The first part is called the prior distribution. It contains all information that you have about the parameters, before actually looking at the data. The second is called the likelihood. It contains the unnormalized likelihood of seeing exactly the data that you saw under all possible explanations of the data. The third part, the denominator, can either be thought of as the subjectively expected value of the likelihood function or as the marginal probability of the data.

It is the prior that is going to concern you here. You multiply the likelihood times the prior. The prior for the method of maximum likelihood for the normal distribution with known variance is the uniform distribution.

Prior to seeing the data, every possible value for the population mean is equiprobable. You have not seen the data. For the method of maximum likelihood, it is like multiplying the resulting normal distribution by one. All the information comes from the likelihood.

The Bayesian may not do that. For example, imagine that you were calculating the number of calories in a new variety of green beans.

You know negative calories do not exist, so you apply zero prior weight there. You also know that it is closely related to two different varietals. You know the calorie estimates for both them very well. You believe your calorie count will be near the calorie count for the two varietals. So you give very high prior weight in the region estimated for the two varietals and drop the probability close to zero outside that region.

The effect is to vastly narrow the likely location by giving great weight to probable regions and nearly zero weight in improbable regions and no weight in impossible regions, such as -5 kcal.

That is what a shrinkage estimator is doing. For any Frequentist shrinkage estimator, you can find a Bayesian likelihood and a proper prior that will produce it. The statistic would be extracted by applying a loss or utility function to that posterior distribution.

Of course, Frequentist statistics don't have a prior distribution, so you get something that acts quite often like either a weighting scheme or a mechanism to pull estimates together.

It should be noted that the measurements do not have to be about related things.

So, for example, Stein's original estimator is equivalent to taking the grand mean of a set of things to estimate and using that as an empirical prior distribution to estimate the individual vector of population means of interest.

Since a covariance is just an expectation, you are doing the same thing, but in the realm of covariance.

A way to think about it is that you are extracting more information from the grand mean than was going to be present in the separate sample means. However, that creates some weird intellectual issues.

Imagine you were estimating the price of diamonds in Hong Kong under certain grading restrictions, the price of pork futures for delivery at a particular location on a specific time and date, the price of gold futures, and the height of third-grade boys and girls in Muncie, Indiana.

Stein's lemma says that the estimated things do not have to be related for you to improve your estimate. Any real set of random things will do. That is for any level of an estimator, not just the mean. It also goes for the covariates if you were interested in them as well.

In a sense, shrinkage estimators use data to make a first pass guess at the general location of the population means. The shrinkage estimator will shrink a lot if that is true and very little if that first pass is not true.

That brings us to the maximum likelihood estimators. They are invariant to monotone transformations of the data. If the sample mean is 8 and you take the log base 2 of the data, then the sample mean will be 3. That is not true for the MVUE or Bayesian methods.

Shrinkage estimators are not invariant. If you take the log of the data, then you will change the estimator.

Likewise, they are not unbiased, so you are including principled bias that can add information, but you are biased.

The linkage to admissibility is that a shrinkage estimator will stochastically dominate a regular non-Bayesian estimator as well. The mean squared error of a shrinkage estimator will be less than for a standard non-Bayesian estimator everywhere.

Because Bayesian estimators are automatically optimal estimators under admissibility, it is not a meaningful criterion to judge comparative tools. You cannot compare a Bayesian method in a straightforward way with a non-Bayesian method. The Bayesian method cannot lose even if it does not look like the shrinkage estimator. It can tie, but it cannot lose.

It is important to note, however, that it cannot lose even when it actually is a worse estimator if real proper priors are used.

To understand why that might be the case, consider the prior beliefs of those that believe the 2020 election was stolen or that believe that Ivermectin is superior to one of the vaccines or monoclonal antibodies. As long as their prior distributions are not degenerate distributions, that is to say, that they attribute a one hundred percent certainty to their ideas, then they will be admissible. In some cases, even those with perfect certainty would be admissible. Admissibility is a poor standard to judge a Bayesian method against a non-Bayesian one.

Under scoring standards, post hoc, some posterior estimators will be poorly calibrated.

It is simple but expensive and involved to prove that the 2020 election results are the correct results. Likewise, there is enough data on Ivermectin to exclude it as a primary treatment, although it is being investigated to determine if it makes other treatments work better.

People who believe that Ivermectin works as a treatment or as a prophylactic drug are poorly calibrated with reality but their estimate is probably admissible. I would have to either look up what estimators are admissible under the likelihood or do the calculation myself. I am not that motivated to do either as it is not really relevant, I hope.

The same is true for people that believe Trump won the 2020 election.

Under admissibility, both poorly calibrated results win over better calibrated results.

Covariance shrinkage estimators provide some weighting scheme that may or may not be a good weighting scheme. The weighting could be something like Stein's estimator or something like ridge regression. There are covariance specific shrinkage estimators.

You are taking the data and using it twice to try and extract just a little bit of extra information out of it. On average, a shrinkage estimator will produce a better result than either the MVUE or the MLE for higher dimension problems, $k>2$, other things constant.

An example of the weighting a prior has on the likelihood to produce the posterior distribution is shown here.

In this one-dimensional example, that is what shrinkage is doing, except that the prior is being created by extracting information from a parameter that you are not interested in, to get a better estimate about one that you are interested in.