How to cluster ETFs to reduce cardinality for portfolio selection

Question

I'm looking to run portfolio optimizations using various optimization goals - e.g. minimum variance, max diversification etc. My challenge is if I want to do this on ETF's which ones do I pick to run the optimization on?

Say there is a universe of 200 or so ETF's - is there some form of clustering I can do to reduce this down to a smaller set of 20 or so to optimize? Or is this best handled by letting the portfolio optimizer itself apply the appropriate weights out of the larger set?

What techniques should I consider for clustering - i.e. what metrics make sense is it correlation, mean return (I doubt it since that's so noisy), anything else?

---

Updates:

To clarify what I'm trying to do is whittle down the 500 names in the SP500 to 20 or so clusters and then from each cluster take the most representative stock to get 20 names. I would then do portfolio opt on the 20 names Following a series from the amazing systematic investor blog I've been able to get really nice results doing some clustering as follows:

# Try various clustering schemes on xrets (log returns matrix)
xrets.scaled <- scale(xrets)
xrets.euclid.dist <- dist(t(xrets.scaled))

xrets.correlation.distance <- as.dist(1-cor(xrets))

fit <- hclust(xrets.correlation.distance, method="ward")
plot(fit)
k.numclusts <- 30
groups <- cutree(fit, k=k.numclusts)
rect.hclust(fit, k=k.numclusts, border="red")

However all this gives me is clusters e.g. in the SP500 it found clusters of the following form (illustrating just one cluster):

> CAG : ConAgra Foods Inc. : [1] NA CL : Colgate-Palmolive Co. : [1] NA
> CPB : Campbell Soup Co. : [1] NA GIS : General Mills Inc. : [1] NA HNZ
> : H. J. Heinz Company : [1] NA HRL : Hormel Foods Corp. : [1] NA K :
> Kellogg Company : [1] NA KMB : Kimberly-Clark Corporation : [1] NA KO
> : The Coca-Cola Company : [1] NA MKC : McCormick & Co. Inc. : [1] NA
> PEP : Pepsico Inc. : [1] NA PG : Procter & Gamble Co. : [1] NA SJM :
> The J. M. Smucker Company : [1] NA

This still doesn't tell me though which stock in each cluster is the most optimal to select for further portfolio opt. Any ideas?

Answer 1

First, find out which ETFs are correlated with one another over time. Let the data matrix $\mathbf{X}$ of ETF price returns have $t$ rows and $p$ columns, where the $t$ rows are bars or days, and the $p$ columns in the dataset are ETFs. Next, determine the correlation matrix $\mathbf{R}$ for the ETF-to-ETF correlation, and then run principal components analysis (PCA) to identify which ETFs load or are correlated with each principal component (PC). To understand PCA, if you have a correlation matrix $\mathbf{R}$ for e.g. 50 ETFs, then PCA will create 50 PCs which have zero correlation between each other. For each ETF, there will be a loading value (i.e., correlation) for each PC. If the price returns of several ETFs are correlated, they will likely load on the same PC. Commonly, loading values (correlation) of 0.45 and greater are considered to be large enough to warrant further investigation.

You can use the loadings of each ETFs on the various PCs to group together ETFs whose price returns are correlated. For example, bank ETFs may correlate (loading>0.45) with the 1st PC, oil ETFs with the 2nd PC, emerging growth ETFs on the 3rd, and so on.

Assets (ETFs) commonly don't always correlate with everything else, but will correlate with assets in the same sector. PCA creates artificial vectors (eigenvectors) which have zero correlation with one another, and therefore by observing which ETFs correlate mostly (>0.45) with a given PC, you can essentially group.

PCA is wholly linear, and there are non-linear methods such as Laplacian Eigenmaps, Local Linear Embedding, Locally Preserving Projections, Diffusion Maps, etc. which can be employed. Another term for the non-linear techniques is "distance metrics learning."

Answer 2

I would look to run a pre-optimization routine over the whole universe of 200+ ETFs. I would use this pre-optimization to reduce the universe to a cardinality that provides optimal diversification effects. You can do that by first looking at pair-wise correlations and then also run optimizations to reduce portfolio variance by utilizing the covariance matrixes. In that way you will already filter out highly correlated assets that are useless to combine in a portfolio. You could use additional constraints such as minimum liquidity requirements to decide which assets to keep and which to toss out. But in any case, try to eliminate highly correlated assets and those assets that do not reduce portfolio variance in a meaningful way.

The following may help doing that:

http://www.rinfinance.com/RinFinance2009/presentations/yollin_slides.pdf

https://www.rmetrics.org/files/freepdf/PortfolioOptimizationSample.pdf

https://ccnet.stanford.edu/cgi-bin/course.cgi?cc=msande242h&action=handout_download&handout_id=ID131921138820714

Answer 3

When I select assets for a portfolio given an universe, I tend to pick ones that span the beta spectrum, given your selected benchmark. I find that if your portfolio of assets have varying volatility or correlation, you can achieve better diversification. I didn't come up with the idea but it comes from a rotational system's framework from the link below:

http://cssanalytics.wordpress.com/2010/02/26/rotation-part-3-beta-surfing/

One thing to be careful is to avoid randomly throwing assets in to a optimization procedure as you may find the ending portfolio allocation concentrated with risk that is hidden from a portfolio level. It may be useful to first manually separate the etfs in to their respective categories.

Update: I am not sure the relevance to your research but CSS analytics just started a 2 part post about Cluster Risk Parity which has some ideas that talk about subsetting assets from portfolio universe.

HTH,

Answer 4

I've used the graphical lasso for exactly this kind of thing in the past. You can control the degree of shrinkage, which determines the how tight the clusters become.