How to cluster stocks and construct an affinity matrix?

Question

My goal is to find clusters of stocks. The "affinity" matrix will define the "closeness" of points. This article gives a bit more background. The ultimate purpose is to investigate the "cohesion" within ETFs and between similar ETFs for arbitrage possibilities. Eventually if everything goes well this could lead to the creation of a tool for risk modelling or valuation. Currently the project is in the proposal/POC phase so resources are limited.

I found this Python example for clustering with related docs. The code uses correlations of the difference in open and close prices as values for the affinity matrix. I prefer to use the average return and standard deviation of returns. This can be visualised as a two dimensional space with the average and standard deviation as dimensions. Instead of correlation, I would then calculate the "distance" between data points (stocks) and fill the affinity matrix with the distances. The choice of the distance function is still an open issue. Is calculating the distance between data points instead of correlations valid?

If it is can I extend this approach with more dimensions, such as dividend yield or ratios such as price/earnings?

I did a few experiments with different numbers of parameters and different distance functions resulting in different numbers of clusters ranging from 1 to more than 300 for a sample size of 900 stocks. The sample consists of large and mid cap stocks listed on the NYSE and NASDAQ. Is there a rule of thumb for the number of clusters one should expect?

Answer 1

You should consider an unsupervised learning algorithm such as K-nearest neighbor ('KNN').

KNN will measure the distance amongst the observations in your space. You can and probably should consider alternative distance functions (besides euclidean) particularly if you are clustering on features such as returns which have outliers. There are quite a few unsupervised clustering algorithms out there - see here. You can certainly include features such as stock characteristics with these algorithms. You can also include the betas of the securities with respect to various risk factors as well. This would allow you to capture the distances in correlation space since a security based covariance matrix can be expressed as the : cross-product of (betas for factors) * covariance matrix of factor returns * transposed(betas for factors).

I would spend time thinking about the appropriate choice of features (which features are stable? which features predict risk or return? which sets of features are contributing unique sources of information? what are the invariants?) and choice of distance function.

Also, if you are mixing features with different unit scales (i.e. returns, betas, variances) then you need to normalize/pre-process your inputs otherwise the features with the highest variance will be the primary basis for clustering. Alternatively, you can stick to one class of features for your your clustering so you have some more intuition on interpreting the results.

Answer 2

Quant Guy's answer is quite informative for your question already.

Just to add few other things: instead of figuring out the choice of features by your own brain, you could also use machine learning techniques to help in extracting the 'features' for your specific purpose, e.g. risk modeling or returns forecasting or portfolio construction as mentioned by Tal.

Take a look at Principal component analysis and manifold learning (e.g. isomap). Even more interesting, the Unsupervised Feature Learning and Deep Learning. The first two methods both have implementation in the scikit-learn, the library you are currently looking into.

The first two methods mentioned above, could help you not only to extract the more important components from you features, but also to visualize your clustering given your feature dimension is bigger than 2.

Answer 3

Rather than suggesting alternative clustering techniques, as Quant Guy and Flake have (great advice, btw), I'll offer my thoughts on the method you've proposed.

On the characteristics used to cluster stocks: You propose using sample statistics (mean and standard deviation of returns). I would suggest you use the entire return (not price) series. For example, if stock A's returns on two successive days are (+1%,-1%) and B's returns are (-1%,+1%), your method would rank these two very closely based on mean and standard deviation, when in fact they should be quite far apart, particularly if most stock pairs in your sample are positively correlated (which I believe the vast majority are). Potential elaborations on this method include volatility-adjusted returns, market-beta adjusted returns, and excess returns relative to some risk model. I would shy away from using too many non-return characteristics, particularly fundamentals such as dividend yield or P/E, but you may want to introduce size (market cap) and industry. If your sample is exclusively ETFs, I strongly reiterate my advice to shy away from all fundamentals, including size and industry (which are irrelevant for broad ETFs and potentially misleading for narrow ones).

On the distance function: The function referenced in the paper you linked seems very complicated so I can't comment on it, but in general you should consider that if you use more than one type of input (e.g. returns and fundamentals), you should upweight differences which should be close together and downweight differences which are expected to be far apart, using something like Mahalanobis.

On the number of clusters: You will need to think more about your sample to figure out the optimal number of clusters. If you are looking at the entire market, I think statistical techniques are generally good at identifying no more than about 5 independent sources of variation. Assuming each one of these has 2 potential states, that implies $2^5=32$ clusters. If you repeat the analysis using only ETFs in your sample, given all the overlap, I'd go for even fewer, perhaps 10 clusters.

Best of luck. Your question does make sense to me now; the edits helped immensely.