Shrinkage was much en-vogue before random matrix theory (RMT) took everybody's attention in covariance matrix estimation, however the latter also showed its limits. A plethora of other estimators has been presented, but I could not yet spot a golden standard. What is nowadays used most in practice (or what are you using), and why? Also, shrinkage came in different flavours, so I'd like to know which is the favourite among them.

Note that I'm not just asking for the statistical properties of different methods (this would be on Cross Validated in that case), but also their interplay with practical considerations here in the quant world, which might include even non-technical factors.

  • $\begingroup$ I am guessing that RMT = Random Matrix Theory $\endgroup$ – Alex C Jun 8 '15 at 0:54
  • $\begingroup$ Yes, that's it. $\endgroup$ – Quartz Jun 8 '15 at 13:40
  • 3
    $\begingroup$ Do you have any reference for applications of RMT in finance? I have applied shrinkage a houndred times and I have seen people arguing with RMT (that the sample covariance matrix is instable) but not really applying it. $\endgroup$ – Ric Mar 2 '16 at 6:57

I thought I would answer the question of "what am I using." All shrinkage estimators map to a Bayesian estimator that differs only in the prior distributions. In other words, you get a point estimate that is indistinguishable from a Bayesian estimate except that the calculation rule determines the prior distribution. Stein estimators for the Gaussian are nothing more than the ordinary estimator with an empirical prior distribution based off of the grand mean.

Since I usually have real prior information, and so does everybody else, I use Bayesian estimators as I get the shrinkage for free. Bayesian estimators have the nice property that they are "coherent," which means fair gambles can be placed on them, whereas this is not true for Frequentist estimators. In fact, I once did a talk on how you could always game someone using a Frequentist estimator in the right circumstances guaranteeing a sure win no matter how the universe came out.

The downside of a Bayesian estimator is computation time. This is less serious than it appears because Bayesian updating allows you to do the calculations one data point at a time. If you have calculated the solution for every data point up to today, then you can add today's data in and ignore all prior data, reducing the calculation size tremendously. You can do this because yesterday's "posterior density" becomes today's "prior density."

Speed is your enemy here, though. Speed requires compromises if you use a Bayesian method. Although Bayesian predictions are coherent and Frequentist predictions are not, you have to do some compromises or you will not run fast enough if time is of the essence. In particular, the impact of one day's data on the posterior is negligible. Unlike a Frequentist method, you do not really have to update in real time, you can use the data up to yesterday and get nearly identical results. This is because Bayesian methods use the unique information in the data that is not in the prior.

To understand this, imagine you have two data sets and a prior and you merge one data set with the prior to get a posterior density. When you merge the second set into this density, the only changes that will happen in the second posterior density will be from information that is unique to the second set that is neither in the first set nor the prior. Bayes rule ignores redundant information. In essence, this is saying that the unique information content of a single trade is negligible when compared to the joint information content of the set of all trades that have ever happened.

So you do not need to run it in real time unless you believe today is the one day that is unique in the history of trading. You can run with a lag because there is trivial information loss.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.