# Portfolio Theory: Why is so much effort put into the reduction of estimation errors?

In MPT, very much effort by researchers is put into developing methods and techniques to handle the rather poor performance of the estimated means, variances and covariances. There are shrinkage estimations, random matrix theory or approaches that address the optimization constraints, for example. Some of them are very complex and sophisticated and in turn require excessive data processing.

At the same time, all these estimation problems arise from the assumption that asset returns are i.i.d. and moreover normally distributed. It has been shown numerous times that these assumptions are contradicted by empirical evidence. Researchers do not seem to agree on how to classify the probability distribution of asset returns. Some mention the classification as an $\alpha$-stable distribution other than the normal distribution, others suppose to classify asset returns as some non-normal distribution with finite second moments.

My question is: Why don't researchers focus on developing a more adequate distribution of asset returns? I mean, handling the estimation errors seems like approaching the problem from the wrong side to me. Wouldn't it be more promising to find an adequate probability distribution of asset returns and to develop models that implement this distribution? Instead of correcting the errors, one would be able to avoid them from the very beginning. I hope I'm not overlooking something. But from what I've seen so far, the focus really seems to be on estimation error reduction.

I'm just asking out of my own interest, because I've been wondering about this. It's quite astonishing to see the above mentioned assumptions being applied although they have been proven to be inadequate.

Thanks a lot in advance and have a good start into the new year.

• It might be worth knowing that as soon as you make the drift and vol functions (or indeed also stochastic) you're not limited to normal returns. The fact that the second derivative of options gives you the (scaled) probability distribution, and that models are able to accurately replicate all options in a calibration set, gives you the non normal distribution. That being said, you're still not accurately replicating everything though, and some products may be outside of the scope of your model (ie probably variance swaps). Ands that's not even beginning to approach them being iid. – will Jun 15 '17 at 22:21

In MPT investors maximize ex ante expected return for a given level of ex ante variance. Gaussian-ity or iid-ness of returns are not requirements.

The problem is estimating these ex-ante quantities using ex-post transformations of the recent history. Many of the sophisticated techniques that you mention try to offer robustness of ex post estimators in the presence of fat tails and/or lack of iid-ness.

Researchers have correctly (imo) concluded that trying to identify an ever changing 10000-dimensional distribution is futile, therefore it makes sense to try and focus on the quantities of interest only.

• But the estimators being used are unbiased only for i.i.d. random variables. So applying these estimators requires the respective random variable to be i.i.d. Moreover, it is well known that the Markowitz model assumes normal distribution of returns. Providing robustness seems only partly reasonable to me. Because you try to alleviate a problem that could be addressed from the very beginning (applying more suitable estimators, describing asset returns with a more suitable probability distribution, etc.). – WiWiStudent Jan 1 '17 at 15:08
• Well known that Markowitz assumes normal distribution of returns? Can you please point out where: math.ust.hk/~maykwok/courses/ma362/07F/markowitz_JF.pdf Markowitz only requires 'anticipated expected return' and 'anticipated variance', which I call ex ante. He does not prescribe how this anticipation is formed or under which assumptions it is formed. – Kiwiakos Jan 1 '17 at 16:35
• In fact he makes two remarks: "[i]n a general presentation we must recognize that the probability distribution of yields of the various securities is a function of time" and "[t]his paper does not consider the difficult question of how investors do (or should) form their probability beliefs". – Kiwiakos Jan 1 '17 at 16:43
• You´re right. I have to be more specific. In fact, Markowitz did not assume any specific probability distribution or that asset returns are i.i.d. But to implement the model in practice, the mean, variance and covariances have to be estimated. And for the sake of simplicity, it is almost always assumed that asset returns were i.i.d., since it is possible to estimate the parameters via their sample representations in this case. But the question remains. Why not focus on the derivation of better estimators and more suitable assumptions when implementing the Markowitz model? – WiWiStudent Jan 1 '17 at 18:53

There are two ways to answer your question. One is direct and has less depth to it, the other is more indirect and has a lot of depth to it. I will begin with the indirect one because it has wide ranging applications beyond finance or economics and because it should serve as a warning to journal editors and so forth. Also, it covers an area of statistics that everyone but statisticians have forgotten about.

Although the idea of a statistic is rather old, the field of statistics is rather new. It is probably the newest or nearly newest of all fields. Aeronautics is older. Genetics is older. It opened up a ton of practical questions that it took time to solve. It has to do with how the field defined as statistic. A statistic is any function of the data. This means that almost every statistic is useless as there are uncountably many functions.

This led to a process to decide which statistics to keep and which to discard. This created unexpected results. If you are finding poor estimators as a result of your theory, then there is a good chance, you are doing it wrong. In the defense of finance, it has been struggling with this since Mandelbrot published the first empirical refutation of mean-variance finance. It tried to solve it in the 1960's but a couple of things got in the way. The first was the use of punch card technology. Even if work by Eugene Fama or Mandelbrot were correct, it would have resulted in problems that would take decades to solve. The second was that there was no reason for them to be correct. There was no theory behind the observations.

The unexpected result, in searching for a statistic, was that all Bayesian statistics were admissible. This was surprising because it was proved with Frequentist axioms. It also found that all other statistics were valid to the extent they either mapped to a Bayesian measure in a particular case or at the limit. It provides a test, however. If you can stochastically dominate a measure, then you drop that measure. If you hunting for accurate measurements that work, then something deeper is going on and you are missing it.

The more direct answer is that the distribution of returns, for the Markowtiz model to be correct, have to have certain properties. The first is that there needs to be a mean in order to have an expectation in the first place. Most standard distributions have a mean, but not all do. The Cauchy distribution and, in general, the Paretian distributions of Mandelbrot's article do not. The Cauchy distribution is $$\frac{1}{\pi}\frac{\sigma}{\sigma^2+(x-\mu)^2}.$$

The second is that if a mean exists, a covariance matrix needs to exist. Not all distributions with a variance have a covariance in its multivariate form. The hyperbolic secant distribution is an example of that. It is $$sech\left(\frac{x-\mu}{\sigma}\right).$$

There have been attempts to use both in empirical finance. If either of those distributions are present in the likelihood function, then mean-variance finance is indefensible. The former is problematic because you cannot form an expectation on your returns in the first place. They are excluded by the laws of general summation. The second is a bit more subtle because if the second is present none of the assets can be independent, but none of them can covary either. They can comove, but not covary. It creates a very ugly issue.

There is a paper that derives the distribution of returns at https://ssrn.com/abstract=2828744. It shows that there are many distributions that can be present. The logic of the paper is that returns are not data, rather, prices are data. Returns are transformations of data. In particular, they are the ratio of jointly distributed variables, a present value and a future value. The distribution depends upon the rules in use to create the prices. As a result, stocks have different returns than antiques because the auction process is different.

As it happens, all distributions for equity securities include some mixture of a transformation of the Cauchy distribution. Because the distributions involved lack a sufficient statistic, any point estimator has to lose information, so no non-Bayesian solution exists for projective problems (such as choosing an allocation), and should be avoided for inferential questions if possible. You cannot avoid them in your true hypothesis is a sharp null hypothesis as there is no good Bayesian solution for sharp null hypotheses.

A population test of the paper can be found at https://ssrn.com/abstract=2653151

There are also papers to replace the method of pricing options and the rules of econometrics. Papers to create optimal portfolios and to extend stochastic calculus are in process. The distributions paper will be presented at the Southerwestern Finance Association Conference in March.

Some things will have to change. You cannot make an assumption of i.i.d. variables, for example. The entire discussion of the Solow convergence will have to change in economics and so the core of the whole discussion of capital, physical, financial and human.

A lot of focus will end up on the scale parameter. In the Cauchy distribution, there is no covariance matrix. If you had a one asset portfolio, denoted $a$, then it may have a scale parameter $\gamma_a$. If you switch to a two asset portfolio you do not get two scale parameters, let alone a covariance style matrix. Instead you get a new scale parameter $\gamma_{ab}$. If you got fancy and used a vector process, all the vectors would jointly share a scale parameter $\gamma_v$. Taking the logarithm brings you to the hyperbolic secant distribution and so no gain is had. It also has no covariance matrix, but OLS does. OLS would be measuring something that does not exist.

• Thank you very much for this extensive, insightful answer. Let me try to outline and summarize some key points you made (please correct me if I´m wrong). 1) The critical assumptions of i.i.d. random variables and normal distributions arise from the implementation of the Markowitz theory. 2) These assumptions are not supported by empirical evidence. It is likely or certain that returns are to some extent distributed in a way that creates issues (e.g. non-existence of mean, variance or covariance). 3) Research continues but will create problems that won´t be solvable in the near future. – WiWiStudent Jan 3 '17 at 13:29
• Some things are readily solvable in the near term. Except for mergers, bankruptcies and very illiquid securities, the truncated Cauchy distribution is a close match to reality. This ignores the budget constraint, but the effect is small. Thus, you can do regression on the general case. If you factor in liquidity explicitly, you can solve anything except mergers, easily. Mergers are a headache and exceed a posting here. Because none of the distributions have a sufficient statistic, it leaves you with Bayesian methods alone. (continued) – Dave Harris Jan 3 '17 at 18:08
• (continued) So lets do a simple example. Let $x$ be the earnings to price ratio and $i$ be the interest rate. Let us assume you want to predict returns exclusive of dividends (because that just makes the problem really long) where $r=\beta_xx+\beta_ii+\alpha+\epsilon$, then your likelihood function is simply $$\frac{\sigma}{\sigma^2+(r-\beta_xx-\beta_ii-\alpha)^2}$$. Use Metropolis Hastings and you have a return estimation model, not necessarily an intrinsically good one, I can think of better easily, but you have one. – Dave Harris Jan 3 '17 at 18:12
• You can go to ssrn.com/abstract=2653255 and you have an options pricing model. In fact, because you are nearly forced into Bayesian methods, you have an entire field built for you already called Bayesian Decision Theory. Bayesian Decision Theory is optimal for finance anyway. Its been using Frequentist Decision Theory, but that is not coherent so you can always game a market maker if the market maker uses Frequentist theory anyway. Bayesian methods are intrinsically coherent so market makers cannot be gamed, that is, they cannot take a sure loss. – Dave Harris Jan 3 '17 at 18:19

Just to provide some background on why mean-variance is so sensitive to estimation error, consider an unconstrained mean-variance optimization. The optimal weights are given by $$w=\frac{1}{\lambda}\Sigma^{-1}\mu$$ where $\lambda$ is a risk aversion coefficient, $\Sigma$ is the covariance matrix, and $\mu$ is a vector of returns.

We can examine the sensitivity of the weights with respect to the parameters, by calculating the partial derivatives. The most relevant is $$\frac{\partial w}{\partial\mu}=\frac{1}{\lambda}\Sigma^{-1}$$ The practical nature of the covariance matrix is that it will typically have very high values when inverted. As a result, this means that a small change in $\mu_{i}$ will have a large impact on all $w$. One can do the same analysis with variances and covariances, but it is less of an issue.

The problem of estimation error is that classical mean-variance optimization is very sensitive to return forecasts.

The question then is why focus so much on estimation error when the problem is the return models. In some sense, shrinkage models and random matrix theory are actually an attempt to create better models. They are creating models in recognition that classical mean-variance is susceptible to estimation error.

The question suggests non-normal models. While there is value in non-normal models, they are not a solution for estimation error. If the techniques that reduce the impact of estimation error rely too heavily on normal distributions, then tweak the assumptions and see if you can make them work with non-normal distributions.

Most importantly: if you have a hammer, try not to hammer your finger.