How do I find the most diversified portfolio, or least correlated subset, of stocks?

I have a trading system that chooses top 10 stocks in Nasdaq 100 ranked on relative strength and some other factors. However, I'd like to take positions in only 5 of these 10 stocks based on how minimally correlated these are to others for diversification effect. How do I resolve this? I do have the correlation/covariance matrices computed. Literature seems to indicate applying weights to reduce correlations but I felt there should be a simpler solution. That said, the stocks doesn't need to be equally weighted if it is easier to compute these weights.

A computationally easier solution is preferred even if it is not completely accurate since I need to implement this in Amibroker trading software.

-
Weighing by 1-correlation is already a pretty simple method. Could you elaborate on why that's difficult? –  chrisaycock Aug 18 '11 at 2:38

If you only need to pick 5 out of 10 and want equal weights then just enumerate all 252 possibilities (as pointed out above) and compute the portfolio volatility

$(\textbf{1}'K^{(i)}\textbf{1})^{1/2} = \left( \sum_{ij}K^{(i)}_{ij} \right)^{1/2}$,

where $K^{(i)}$ is the covariance matrix for the $i$th subset. Then use whatever subset gives the lowest portfolio volatility. Here you are minimizing portfolio volatility so you will be biased towards lower volatility stocks. If you don't care about volatility per se and just want to minimize the contribution to portfolio risk related to correlation (somewhat loosely defined) then you can use the Most Diversified Portfolio (MDP) method. This method aims to minimize the diversification ratio

$\frac{w'\sigma^{(i)}}{\left(w'K^{(i)}w\right)^{1/2}} =\frac{\sum_j\sigma_j}{ \left( \sum_{ij}K^{(i)}_{ij} \right)^{1/2}}$

Again, just plug in values for each subset and use whatever gives the largest value.

Personally, I would argue that a few aspects of what your doing are inefficient.

1. Why equal weights? If you have a covariance matrix then you can almost always find less risky portfolios. Each stock has a different volatility so equal weights tends to take too much risk in more volatile stocks.
2. Why consider only your top 10? It is possible your best 5-stock portfolio includes stocks outside of your top-10 rankings due to correlations.
3. Instead, consider attempting to generate expected returns for your stocks. You can do this by running a simple linear regression using your sorting metric.

As has been pointed out, the full mean-variance optimization is hard to solve when you have a cardinality constraint and a large number of stocks to consider. A common approach is to employ $l_1$ norm based methods. The gist of it is instead of solving the standard mean-variance QP

$\min_w \{ \lambda w'Kw - r'w \}, w \geq 0, \sum_i w_i = 1$,

drop the budget constraint and add an $l_1$ penalty, i.e.

$\min_w \{ \lambda w'Kw - r'w + \gamma ||w||_1 \}, w \geq 0$.

As you slowly increase $\gamma$ the $w$ vector will get sparser. Stop once you only have 5 non-zero values. Afterwards, re-scale the weights to sum to one. This version of the problem is a convex relaxation of the actual cardinality constrained problem. The $l_1$-norm penalty can also be motivated as the solution to a robust portfolio optimization problem where returns are uncertain, but satisfy a box constraint.

-

One simple method, based on the principles of mean-variance optimization, is to set the weights proportional to the product of the inverse of the covariance matrix and a vector of standard deviations. This implicitly assumes that the normalized expected return of each stock is equal. If you wish, you can take only the top 5 weights and set the others to zero. The actual problem you face, of selecting just 5 stocks, can be solved rigorously with an optimizer, but since it is not a quadratic program, may be difficult to solve.

Update

A more sophisticated but very interesting additional possibility is to find the "Maximum Diversification Portfolio (MDP)", as defined in Toward Maximum Diversification (free version, hat tip vonjd). The MDP is defined as the portfolio that maximizes the Diversification Ratio (DR), which in turn is defined as the ratio of the portfolio’s weighted average volatility to its overall volatility. A follow-up paper investigates the properties of this portfolio. From the paper:

This measure [DR] embodies the very nature of diversification whereby the volatility of a long-only portfolio of assets is less than or equal to the weighted sum of the assets' volatilities. As such, the DR of a long-only portfolio is greater than or equal to one, and equals unity for a single asset portfolio. Consider for example an equal-weighted portfolio of two independent assets with the same volatility: its DR is equal to $\sqrt{2}$, and to $\sqrt{N}$ for $N$ independent assets.

$DR(\bf{w})=\frac {\sum_i{\it{w}_i\sigma_i}} {\sigma(\bf{w})}$

-
You could probably use DEOptim() R package to solve this complex objective function. Along the lines suggested, I would add a constraint for the max number of assets. Also, you can include in the objective function a vector corresponding to the expected returns of each stock. Since you are optimizing only over 10 stocks the algorithm would converge rapidly. –  Quant Guy Aug 24 '11 at 22:43
@QuantGuy I haven't tried this, but the paper suggests that a long-only MDP will typically have much fewer assets than the total available, so a cardinality constraint may not be necessary here. –  Tal Fishman Aug 25 '11 at 1:44
@TalFishman I've looked at replicating TOBAM's results in the past, and you're absolutely correct: an unconstrained MDP portfolio on the R1K universe typically holds 50-200 names. –  michaelv2 Mar 21 '12 at 14:32
An expanded version of Toward Maximum Diversification from 2011 can be found here: papers.ssrn.com/sol3/papers.cfm?abstract_id=1895459 –  vonjd May 27 '12 at 17:46
This answer doesn't really address the question of selecting 5 stocks. You state "If you wish, you can...", but don't really justify your suggestion. –  MichaelJ Jun 9 at 5:15