How do the in-sample estimates and out-of-sample estimates I so often hear authors refer to in emperical analysis of MVO differ?
The term in sample and out of sample are commonly used in any kind of optimization or fitting methods (MVO is just a particular case).
When you make the optimization, you compute optimal parameters (usually the weights of the optimal portfolio in asset allocation) over a given data sample, for example, the returns of the securities of the portfolio for the past 5 years.
The thing is, if you run your strategy over the given data set (over the last five years), you do it in-sample i.e you evaluate your result over the sample you used to fit it. This should technically give you the best possible result.
If you evaluate the strategy over the next two months, then you will do it out of sample, i.e you evaluate on a period which is different from the one you optimized the strategy on.
In asset allocation, it is important to use out of sample backtesting because you will only be able to invest today the result of an optimization made today. Hence you will earn tomorrow's strategy performance; not yesterday's.
I do not think they are directly applicable to MVO because inherently you always model the efficient frontier or asset selection on in-sample data and the result is measured out-of-sample. You can't say, "hey I model it in-sample over 2005 data and then I measure the performance of the portfolio over 2006 data and compare that with results derived from 2010 data." 2010 returns may be derived from the "most efficient" portfolio that may be completely different from the "most efficient" portfolio you built in 2005.
Actually one inherent weakness of MVO is that it "treats return as a future expectation and uses volatility as a proxy for risk, the flaw being that volatility is a historical parameter and you cannot assume that today’s prices provide an accurate forecast for the future." (Mean–Variance Optimization: A Primer).
Obviously there is connection between in-sample and out-of-sample data as indicated in above quote but as I mentioned I would not consider its usage in the same way as in-sample and out-of-simple data are used, for example in those optimization where the optimization is done once, in-sample, and results are comparable across different out-of-sample time frames.