Why is Markowitz portfolio optimisation so popular considering it is worse than an equal weighted portfolio?

Question

The original paper by Markowitz from the '60s has ~20,000 citations (definitely popular). However several papers I came across show that a $\frac{1}{n}$ asset allocation gives higher Sharpe ratios (not sure about drawdowns) than mean-variance portfolio optimisation (or any subsequent derivative methods). What is the deal with all the attention portfolio optimisation (a more complex method) is getting if its utility seems low in practice?

Answer 1

Markowitz's concepts attracted a great deal of interest from theorists (and still do), but never had much application in practice. The results from practical application were always disappointing (starting in the 1970's, well before DeMiguel, Garlappi, and Uppal (2007) study of $\frac{1}{N}$ portfolios), mainly because it is so difficult to provide accurate estimates of all the parameters involved (especially the expected returns, which in truth no one knows). Nevertheless it was revolutionary because gave rise to other theories, such as CAPM, multi-factor models, Black-Litterman, etc. which eventually we hope will prove useful. Compare investment textbooks before and after Markowitz and you will see that everything changed at that time. Everyone working in (the asset management side of) QuantFinance today is an intellectual descendant of Markowitz, even though they don't use his method in their everyday work.

Answer 2

There has been a split in the community ever since Mandelbrot published his paper "On the Variation of Certain Speculative Prices."

See:

Mandelbrot, B. (1963). The variation of certain speculative prices. The Journal of Business, 36(4):394–419.

To understand why this is so important, you must first realize what economists are trying to do. When you see someone buy an orange, that is the solution to a problem. Economists can only see the solutions to the problems, but the actual problem is what they are trying to study. The goal of economists is to solve "inverse problems."

It appeared that Markowitz in his paper

Markowitz, H. (1952). Portfolio selection. The Journal of Finance, 7(1):77–91.

had come close to a solution. It isn't the CAPM yet, but it seemed as if it were done. What has been missing is that there is a math mistake in Markowitz's paper, but it is very subtle. Indeed, it wasn't until 1958 that mathematicians had actually solved how to do what Markowitz was trying to do and the answer doesn't match. There was a warning by John von Neumann that what appear to be proofs in economics may not be proofs as that branch of mathematics had not been solved yet in 1953. Both of these events were missed in the economic community primarily because the disciplines are a bit stove piped.

For the Fisherian likelihoodist discussion see:

White, J. S. (1958). The limiting distribution of the serial correlation coefficient in the explosive case. The Annals of Mathematical Statistics, 29(4):1188–1197

By the 1970's empirical falsifications of the CAPM were beginning to roll out and in the 21st century, they were cataloged. For the CAPM you can find them at:

Fama, E. F. and French, K. R. (2008). Dissecting anomalies. The Journal of Finance, LXIII(4):1653–1678.

and for the Black-Scholes model, you can find it at:

Yilmaz, B. Z. (2010). Completion, pricing, and calibration in a levy market model. Master’s thesis. The Institute of Applied Mathematics of Middle East Technical University.

Because the math mistake in Markowitz and subsequent papers was missed and because economists have to solve the inverse problem, which is not something finance professionals must do, a tremendous emphasis has been placed on the CAPM and its children like the consumption CAPM or its grandchildren such as the Fama-French model.

To be very fair to Dr. Markowitz, he tested his own theories empirically at:

Markowitz, H. and Usmen, N. (1996a). The likelihood of various stock market return distributions, part 1: principles of inference. Journal of Risk and Uncertainty, 13:207–219.

and

Markowitz, H. and Usmen, N. (1996b). The likelihood of various stock market return distributions, part 2: empirical results. Journal of Risk and Uncertainty, 13:221–247.

In communication with him, he has been a bit more skeptical of his own work than those that followed him. The continuous time content seemed even more worrisome to him. He performed tests in the 90's and were quite rigorous in their methodology.

Markowitz and Usman almost solve the issue in their paper, but they make a mental slip that catches them up. They took the log approximation of returns and not returns themselves. This is a very big deal because this is a transformation that has very different properties from those of the raw data. It changes the statistical distribution. I believe they would have caught what was really happening had they not taken the approximation. They came very close. There wasn't a mental recognition that they had triggered a math alteration in doing that and so they interpret the data as implying that stocks follow Student's t-distribution with 2-3 degrees of freedom. This isn't quite correct, but it is close. In log space, it follows a hyperbolic secant distribution. See, for example,

Harris, D.E.(2017) The Distribution of Returns. Journal of Mathematical Finance, 7, 769-804.

None of the CAPM models can be valid because, as mentioned above, there is a math mistake in them. In fact, the Sharpe ratio is mathematically invalid as well except when solved with logs, although it isn't clear what it means in log space. It is intimately related to Student's t-test, but there is an argument at

Harris, David E., Why Practitioners Should Use Bayesian Statistics (January 25, 2016). Available at SSRN: https://ssrn.com/abstract=2656681 or http://dx.doi.org/10.2139/ssrn.2656681

that for the general case, there does not exist an admissible non-Bayesian estimator for equity securities. Before using it, one should carefully walk through the implications of the math. I have not done that for the Sharpe ratio. It may nor may not be an admissible statistic.

The CAPM still receives a lot of research because the fact there is an error in it is not well known. It is also because the error can be expressed in a number of different ways depending upon the axiom system you are using. For example, if you are using the Fisherian system of thinking called Likelihoodism then rather than there being a math error in the equations, you can show that the theory could never be tested. That is to say, no test statistic could ever be created to determine if it is true or false.

The question then becomes is something valid science if no method of falsification can exist. On the other hand, in the Bayesian framework, you can easily show a mathematical contradiction is created by the model assumptions.

The short answer is that people use the CAPM because they still need to solve the inverse problem. A partial solution can be found for option pricing at:

Harris, David E., Pricing European Style Equity Options (August 30, 2015). Available at SSRN: https://ssrn.com/abstract=2653255 or http://dx.doi.org/10.2139/ssrn.2653255

The CAPM is an attempt to invert the problem. As there is so much groundwork left to do, it isn't clear what people should be using yet. My suggestion is to begin at:

Parmigiani, G. and Inoue, L. (2009). Decision Theory: Principles and Approaches. Wiley Series in Probability and Statistics. Wiley, Chichester, West Sussex.

EDIT Per a request, I have edited the response. Because there are multiple Bayesian and non-Bayesian axiom systems involved there are multiple possible ways to answer this question.

Remembering that the residuals and raw data are leptokurtic and that the math is mesokurtic, the question is why would this be the case. In particular, it should not matter that the data is leptokurtic because the sample is so large that the residuals should long since have become mesokurtic. This follows from the central limit theorem. While it is true that millions of data points are far from infinitely many data points, the results are surprising unless there is no variance.

To explore this question it is important to go earlier into optimization theory than most people would do and to discuss some ideas that are usually ignorable. First, we need to discuss the Bayesian likelihood function and the non-Bayesian density function. If $$\frac{1}{\pi}\frac{1}{1+(x-\mu)^2},\forall{x}\in\chi$$ is your density function, where each draw is i.i.d. then your Bayesian likelihood function is $$\frac{1}{\pi}\frac{1}{1+(x-\mu)^2},\forall{\mu}\in\Theta,$$ where $\chi$ is the sample space and $\Theta$ is the parameter space.

For a variety of reasons, this will become important. The next question is how to estimate $\mu$. For the Bayesian this question is simple, you use Bayes theorem over the entire parameter space. For the Frequentist or Likelihoodist the question is far less simple. The general algorithm for either, though, is to generate a sample statistic that estimates the population parameter and then to develop a test that can allow the performance of inference. Although the rule generation process and the interpretation of the two primary non-Bayesian methods are different, they roughly share the same ideas.

The definition of a statistic is any function of the data. While the mean, median and mode of a sample is a statistic, so is the sum of the cosines of the data. There has to be a method to determine which one to use. At the beginning of the field of statistics, Abraham Wald created a set of decision rules that would determine which estimators are admissible and which are inadmissible.

For data drawn from a Gaussian distribution, the sample mean is admissible, but the sample median and mode are not. Neither is the sum of the cosines of the data points. There is more though here. Wald, a Frequentist and using Frequentist axioms, determined that all Bayesian estimators are admissible. Further, non-Bayesian estimators are admissible under only two circumstances, when they map at every sample to a Bayesian estimator or they converge to the same value at the limit.

I point this out because it will save us time to note that the Bayesian rules will always be optimal in the sense that you can never stochastically dominate them, and the Frequentist and Likelihoodist only inherit optimality when that optimization maps to the Bayesian. This means I can solve this just one time and not sin too badly.

Now let us go back to the very beginning of mean-variance finance to the intertemporal budget constraint $$\tilde{w}=R\bar{w}+\epsilon.$$ Since this is investing, you want to make a profit, so $R\ge{1}$ and generally you can say $R>1$.

Mann and Wald in 1943 were able to show that the maximum likelihood estimator for $R$ is always the least squares estimator for any distribution of $\epsilon$ that was centered on zero and had a finite variance that is greater than zero.

In Frequentist theory $R$ is known as it is the null hypothesis and $\epsilon$ is not known. In Bayesian theory, $R$ is unknown, but $\bar{w}$ is fixed and was not drawn from a random sample. It is important to realize that data are no longer random. All kinds of problems that happen in the sample space are gone now because the only segment of the sample space you care about is the sample that was actually observed.

This leaves three estimation cases, $w_{t+1}=Rw_t+\epsilon_{t+1}$, $R_t=\frac{w_{t+1}}{w_t}$ and $r_t=\log(w_{t+1})-\log(w_t)$. For our purposes, we are going to stick with the first case, though the second and third cases are far more general and would take way too long to solve here.

From White's paper above you can derive both the estimator and the test statistic for R for any distribution for epsilon including jump diffusion models and so forth. The test statistic is always the Cauchy distribution and the maximum likelihood estimator is the least squares estimator, which is a form of sample mean. The Cauchy distribution is $$\Pr(x|\mu;\sigma)=\frac{1}{\pi}\frac{\sigma}{\sigma^2+(x-\mu)^2}.$$ The expectation of the Cauchy distribution does not exist and so the estimator cannot be tested.

Now there is a very interesting relationship between the sample mean and the data from a Cauchy distribution. If $$\Pr(x|\mu;\sigma)=\frac{1}{\pi}\frac{\sigma}{\sigma^2+(x-\mu)^2},$$ describes your data, then $$\Pr(\bar{x}|\mu;\sigma)=\frac{1}{\pi}\frac{\sigma}{\sigma^2+(\bar{x}-\mu)^2}.$$ It follows that the distribution of $R$ is the Cauchy distribution. Since it has no expectation, then E(R) does not exist. Hence the Markowitz optimization is defeated in a Bayesian environment. No one can ever form an expectation about a return. Since the Bayesian estimator is defeated, all non-Bayesian estimators are defeated.

It is important to note that the estimator only converges to the Cauchy distribution in the form of the intertemporal budget constraint. It becomes an ugly mixture distribution in the other cases. It's also important to note in the log case, that the hyperbolic secant distribution has no concept of covariance. Two assets cannot covary, they can comove, but they cannot covary. Indeed, it is also impossible for them to be independent of each other because $$f(x_1)*f(x_2)\ne{f(x_1,x_2)}.$$

The contradiction is that you want to make a profit AND you are asserting the existence of an expectation and a variance-covariance relationship. These are mutually exclusive.

Hopefully final edit The original question was split into two parts. The first is why is a non-optimal portfolio outperforming an optimal portfolio and secondarily, if this is the case why spend so time on Markowitz style models.

The former is simple. Except for very narrow cases involving bond-only portfolios, the assumption of a profitability goal excludes the existence of a mean and therefore a variance. Performing any optimization assuming a mean and a covariance structure is impossible, though as there are large articles as to why, only a summary is provided above. Markowitz style models are unavailable. So when people look at non-optimal portfolios they find they do better than the theoretically optimal portfolios. This is unsurprising because without a population parameter to converge to a statistic is just a random number.

As to why spend so much time, the answer is that everyone knows it and because it appeared to solve the goal of economics, which is to solve the inverse problem.

Everyone just assumed that if you assumed a normal random shock you would get normal returns, but White (see above) shows that you get Cauchy returns with no mean or variance. The problem is that this eliminates all non-Bayesian solutions for general cases because while the Bayesian likelihood function is a minimally sufficient statistic, no sufficient statistic can exist for non-Bayesian methods due to the Pitman-Koopman-Darmois theorem which says none exists for the purposes of creating a projection. A jointly minimally sufficient exists by conditioning on the ancilliary for inference purposes, but not for projective purposes.

Koopman, B (1936). "On distribution admitting a sufficient statistic". Transactions of the American Mathematical Society. Transactions of the American Mathematical Society, Vol. 39, No. 3. 39 (3): 399–409

Indeed, nobody would be discussing tail risk if this process were mesokurtic. I would grab the above bibliography and work forward in time. There are a few things that are missing such as Wald's Complete Class Theorem, but that will get picked up in cited works. You will also want to grab the bibliographies of the works for a more complete discussion.

Answer 3

It is more complicated than that: It is not the optimization per se that leads to inferior results but the data you use.

Kritzman et al. makes a strong case in defense of optimization vs. 1/N in this popular paper:

In Defense of Optimization: The Fallacy of 1/N, Financial Analysts Journal, Vol. 66, No. 2, 2010 by Mark Kritzman, Sebastien Page and David Turkington

Abstract

Previous research has shown that equally weighted portfolios outperform optimized portfolios, which suggests that optimization adds no value in the absence of informed inputs. This article argues the opposite. With naive inputs, optimized portfolios usually outperform equally weighted portfolios. The ostensible superiority of the 1/N approach arises not from limitations in optimization but, rather, from reliance on rolling short-term samples for estimating expected returns. This approach often yields implausible expectations. By relying on longer-term samples for estimating expected returns or even naively contrived yet plausible assumptions, optimized portfolios outperform equally weighted portfolios out of sample.

Another effect is that as soon as you diverge from cap weighted portfolios you automatically add other factors to the mix. In the case of equal weight you especially add size and value factor loadings.

More on this can be found in the following fascinating paper:

The Surprising Alpha From Malkiel’s Monkey and Upside-Down Strategies, The Journal of Portfolio Management, Summer 2013, Vol. 39, No. 4: pp. 91-105 DOI: 10.3905/jpm.2013.39.4.091 by Robert D. Arnott, Jason Hsu, Vitali Kalesnik, and Phil Tindall

Edit
In reality the picture also seems to be different than what is written in academic papers: CXO Advisory Group compared several equal weight ETFs with their capital weighted benchmarks and comes to the conclusion that they underperform their benchmarks on a net basis.

Source: https://www.cxoadvisory.com/31683/size-effect/do-equal-weight-etfs-beat-cap-weight-counterparts/ (behind a paywall).

Answer 4

Markowitz' method for mean-variance optimization was originally intended to demonstrate the "free lunch" of diversification. In that regard, it was and is still very successful. The method gained wider notoriety due its articulation within the Capital Asset Pricing Model (CAPM) (a-la the Sharpe-Black-Lintner Model). The CAPM was really the first application of the arbitrage pricing theory (APT) which could reproduce an assets price given a beta coefficient based on prior mean and variance. It was largely seen as an elegant solution to the efficient market hypothesis (EMH) in which the current market prices represented optimal weights within a market portfolio. The assumption that investors react based on a-priori (or a-posteriori) mean and variance provided the theoretical "glue" for CAPM.

Yet, most tests involving the portfolio optimization fail to outperform many other simpler weighting schemes because they utilize the prior distributions of securities' mean and variance to come up with optimal weights in order to naively maximize the posterior distribution's return per unit of variance (i.e., located an efficient frontier of security weights). If we knew the posterior distribution's mean and variance, we could use Markowitz' method to optimize return in relation to risk. However, it is now an accepted stylized fact that security return distributions are not fixed (i.e. mean and variance are stochastic and are not complete measures of the distribution). Moreover, we now know that the market portfolio's weights cannot be efficient within a mean-variance framework since it has been shown again and again that there other risk factors that influence the posterior distribution.

Fama and French's (FF) entire existence may be predicated as an invective against the CAPM's simple intuition that mean and variance determine the optimal market portfolio. To FF, an efficient market would simply disallow such a simple solution as a single factor linear regression. The key distinction between Sharpe and FF is the view on an efficient market as something which can be priced rationally versus a thing which is too complex for any normative model of investor behavior. Their use of sorts based on size, value, and other factors are proxies for other factors which demonstrate how decisions based on the prior distribution's mean and variance are sub-optimal.

Indeed, FF (1993) found that when portfolios are ranked by size, market beta’s explanatory value falls to zero (and that its utility is not likely salvageable through the remediation of analytical errors).

In my opinion, Markowitz' portfolio optimization techniques have practical value if one can non-naively estimate posterior returns and variances. However, if one continues to use prior returns, one should continue to expect sub-optimal results -- at most, a minimization of excess portfolio variance which is simply a naive side-effect of diversification.