Short of having a 'reasonable' predictive model for expected returns and the covariance matrix, there are a couple lines of attack.
- Shrinkage estimators (via Bayesian inference or Stein-class of estimators)
- Robust portfolio optimization
- Michaud's Resampled Efficient Frontier
- Imposing norm constraints on portfolio weights
Naively, shrinkage methods 'shrink' (of course,no?) your estimates (arrived at using historical data), toward some global mean or some target. Within the mean-variance framework, you can use the shrinkage estimators, for both, the expected returns vector, as well as the covariance matrix. Jorion introduced application of a 'Bayes-Stein estimator' to portfolio analysis. Bradley & Efron have a paper on the James-Stein estimator. Alternatively, you can stick to the global minimum variance portfolio, which is less susceptible to estimation errors (in expected returns)), and use either the sample covariance matrix or a shrunk estimate.
Robust portfolio optimization seems to be another way 'nicer' portfolios can be constructed. I haven't studied this in any detail, but there's a paper by Goldfarb & Iyengar.
Michaud's Resampled Efficient Frontier is an application of Monte Carlo and bootstrap to addressing the uncertainty in the estimates. It is a way of 'averaging' out the frontier and it perhaps is best to read up Michaud's book or paper to know what they really have to say.
Finally, there might be a way to directly impose constraints on the norm of the portfolio weight vector which would be equivalent to regularization in the statistical sense.
Having said all that, having a good predictive model for E[r] and Sigma, is perhaps worth the effort.
Jorion, Philippe, "Bayes-Stein Estimation for Portfolio Analysis", Journal of Financial and Quantitative Analysis, Vol. 21, No. 3, (September 1986), pp. 279-292.
Philippe Jorion, "Bayesian and CAPM estimators of the means: Implications for portfolio selection", Journal of Banking & Finance, Volume 15, Issue 3, June 1991
Robert R. Grauer and Nils H. Hakansson, "Stein and CAPM estimators of the means in asset allocation", International Review of Financial Analysis, Volume 4, Issue 1, 1995, Pages 35-66
Donald Goldfarb, Garud Iyengar: "Robust Portfolio Selection Problems". Math. Oper. Res. 28(1): 1-38 (2003)
Michaud, R. (1998). Efficient Assset Management: A Practial Guide to Stock Portfolio Optimization, Oxford University Press.