# How to cluster stocks and construct an affinity matrix?

My goal is to find clusters of stocks. The "affinity" matrix will define the "closeness" of points. This article gives a bit more background. The ultimate purpose is to investigate the "cohesion" within ETFs and between similar ETFs for arbitrage possibilities. Eventually if everything goes well this could lead to the creation of a tool for risk modelling or valuation. Currently the project is in the proposal/POC phase so resources are limited.

I found this Python example for clustering with related docs. The code uses correlations of the difference in open and close prices as values for the affinity matrix. I prefer to use the average return and standard deviation of returns. This can be visualised as a two dimensional space with the average and standard deviation as dimensions. Instead of correlation, I would then calculate the "distance" between data points (stocks) and fill the affinity matrix with the distances. The choice of the distance function is still an open issue. Is calculating the distance between data points instead of correlations valid?

If it is can I extend this approach with more dimensions, such as dividend yield or ratios such as price/earnings?

I did a few experiments with different numbers of parameters and different distance functions resulting in different numbers of clusters ranging from 1 to more than 300 for a sample size of 900 stocks. The sample consists of large and mid cap stocks listed on the NYSE and NASDAQ. Is there a rule of thumb for the number of clusters one should expect?

-
@Tal: How can I down vote Tal's comment? This is not the first time I see Tal post an offensive comment. Can somebody stop him? I LOVE this site but feel torturing seeing Tal's very subjective and offensive comments. By the way, Navi, I think I know what you try to do and also think that is valid. –  Alchemist Oct 27 '11 at 21:09
@Alchemist let me try to rephrase. Navi, can you please clarify what is the goal of the clustering procedure? For what will you use these clusters? Is it risk modeling, returns forecasting, portfolio construction, etc.? That will make a big difference as to whether it is valid or not. Also, the approach of replacing a correlation (affinity?) matrix with a more general distance function is well founded (see Mahalanobis), although I'm not sure what exactly you are proposing, can you clarify with an equation? –  Tal Fishman Oct 27 '11 at 21:25
@Alchemist: you can't down vote comments, but you can flag them. See the FAQ. –  Ryogi Oct 27 '11 at 23:32

You should consider an unsupervised learning algorithm such as K-nearest neighbor ('KNN').

KNN will measure the distance amongst the observations in your space. You can and probably should consider alternative distance functions (besides euclidean) particularly if you are clustering on features such as returns which have outliers. There are quite a few unsupervised clustering algorithms out there - see here. You can certainly include features such as stock characteristics with these algorithms. You can also include the betas of the securities with respect to various risk factors as well. This would allow you to capture the distances in correlation space since a security based covariance matrix can be expressed as the : cross-product of (betas for factors) * covariance matrix of factor returns * transposed(betas for factors).

I would spend time thinking about the appropriate choice of features (which features are stable? which features predict risk or return? which sets of features are contributing unique sources of information? what are the invariants?) and choice of distance function.

Also, if you are mixing features with different unit scales (i.e. returns, betas, variances) then you need to normalize/pre-process your inputs otherwise the features with the highest variance will be the primary basis for clustering. Alternatively, you can stick to one class of features for your your clustering so you have some more intuition on interpreting the results.

-
I like this answer very much. I already tried different distance functions with differing success. In the article arxiv.org/abs/1110.5847 I linked to a distance function is mentioned. What do you think about this function? Due to limited resources, I need to have a practical performant distance metric. –  Navi Oct 30 '11 at 12:51
Depends on your application. If you are going to find a covariance matrix (which can be expressed in terms of a linear factor model), then using an algorithm that uncovers non-linear structure doesn't help you. However, if you use such an algorithm to group similar individuals and then build, say, different , polynomial regressions or regression models with interaction terms, within each cluster it might be worth using this clustering technique. On its face the algorithm seems interesting - consider your application scenario. –  Quant Guy Oct 30 '11 at 16:29

Just to add few other things: instead of figuring out the choice of features by your own brain, you could also use machine learning techniques to help in extracting the 'features' for your specific purpose, e.g. risk modeling or returns forecasting or portfolio construction as mentioned by Tal.

Take a look at Principal component analysis and manifold learning (e.g. isomap). Even more interesting, the Unsupervised Feature Learning and Deep Learning. The first two methods both have implementation in the scikit-learn, the library you are currently looking into.

The first two methods mentioned above, could help you not only to extract the more important components from you features, but also to visualize your clustering given your feature dimension is bigger than 2.

-
Thanks for the links. PCA forms the basis for spectral clustering doesn't it. I am not familiar with the term "Deep Learning". What does it mean? I have a list about 20 parameters that could be considered for the featureset. Performance wise might be good to do some cherry picking first - I am not sure whether the algorithms in scikit-learn are O(n) or higher order. –  Navi Oct 30 '11 at 13:04

Rather than suggesting alternative clustering techniques, as Quant Guy and Flake have (great advice, btw), I'll offer my thoughts on the method you've proposed.

On the characteristics used to cluster stocks: You propose using sample statistics (mean and standard deviation of returns). I would suggest you use the entire return (not price) series. For example, if stock A's returns on two successive days are (+1%,-1%) and B's returns are (-1%,+1%), your method would rank these two very closely based on mean and standard deviation, when in fact they should be quite far apart, particularly if most stock pairs in your sample are positively correlated (which I believe the vast majority are). Potential elaborations on this method include volatility-adjusted returns, market-beta adjusted returns, and excess returns relative to some risk model. I would shy away from using too many non-return characteristics, particularly fundamentals such as dividend yield or P/E, but you may want to introduce size (market cap) and industry. If your sample is exclusively ETFs, I strongly reiterate my advice to shy away from all fundamentals, including size and industry (which are irrelevant for broad ETFs and potentially misleading for narrow ones).

On the distance function: The function referenced in the paper you linked seems very complicated so I can't comment on it, but in general you should consider that if you use more than one type of input (e.g. returns and fundamentals), you should upweight differences which should be close together and downweight differences which are expected to be far apart, using something like Mahalanobis.

On the number of clusters: You will need to think more about your sample to figure out the optimal number of clusters. If you are looking at the entire market, I think statistical techniques are generally good at identifying no more than about 5 independent sources of variation. Assuming each one of these has 2 potential states, that implies $2^5=32$ clusters. If you repeat the analysis using only ETFs in your sample, given all the overlap, I'd go for even fewer, perhaps 10 clusters.

Best of luck. Your question does make sense to me now; the edits helped immensely.

-