I think the original reference of mean-variance portfolios being “error maximizing portfolios” is:
Michaud, R. (1989). “The Markowitz Optimization Enigma: Is
Optimization Optimal?” Financial Analysts Journal 45(1), 31–42.
The reason is that even small changes in the estimated means can result in huge changes in the whole portfolio structure.
Have a look at this new piece from Andrew Ang which explains this quite well ("4.1 Sensitivity to Inputs", p. 26-27):
Mean-Variance Investing by Andrew Ang
EDIT
For a different perspective see this paper from
Mark Kritzman (2006): Are Optimizers Error Maximizers? Hype versus reality?
From the abstract:
Small input errors to mean-variance optimizers often lead to large
portfolio misallocations when assets are close substitutes for one
another. In fact, when the assets are close substitutes, the return
distribution of the presumed optimal portfolio is actually similar to
the distribution of the truly optimal portfolio. Contrary to
conventional wisdom, therefore, mean-variance optimizers usually turn
out to be robust to small input errors when sensitivity is measured
properly.
A free version can be found on pages 165-168: Here.
EDIT 2
A nice summary of this line of reasoning can be found in
Mark Kritzman (2014): Six Practical Comments About Asset Allocation:
The Myth of Estimation Error:
Cynics often refer to mean-variance
optimizers as error maximizers because they believe that small input
errors lead to large output errors. This cynicism arises from a
misunderstanding of sensitivity to inputs. Consider optimization among
assets that have similar expected returns and risk. Errors in the
estimates of these values may substantially misstate optimal
allocations. Despite these misallocations, however, the return
distributions of the correct and incorrect portfolios will likely be
quite similar. Therefore, the errors do not matter because the
resultant incorrect portfolio is nearly as good as the correct
portfolio.
Now consider optimization among assets that have
significantly dissimilar expected returns and risk. Errors in these
estimates will have little impact on optimal allocations; hence again
the return distributions of the correct and incorrect portfolios will
not differ much. There may be some cases in which small input errors
matter, but in most cases sensitivity to estimation error is more hype
than reality [...]
(Unfortunately I haven't found a free version of the paper - if you find one let me know in the comments and I will update the post).