# What is the textbook answer to dealing with multicollinearity?

I have recently struggled in interviews, for two quantitative trading positions, by producing weak answers to effectively the same (fairly basic) question. I would like to understand, from a quant perspective, what I am missing about multicollinearity.

The question assumes you have a large portfolio of assets (say n=1000 stocks). As I recall, you prepare a covariance matrix (presumably of the price returns). The implication is that many of these returns are correlated. The question basically is, 'what is the problem with this, and how do you solve it?'.

Let $$X\in \mathbb{R}^{m\times n}$$ represent the matrix formed by concatenating vectors of each stock's returns observed over $$m$$ timesteps.

1. My answer to 'What is the problem?':

If the returns are correlated, then there is some 'redundancy' in the matrix $$X$$ (in the extreme case, where a series of returns is identical to another, the matrix is underdetermined). I think that the implication is X is our matrix of features, and we are dealing with a linear regression model. Hence we are worried about the impact of inverting matrix $$X^\top X$$. If we have perfect multicollinearity, then this cannot be inverted; if we just have some multicollinearity, we will fit poorly, giving large errors/instability in our estimates for $$\beta$$.

1. My answer to 'How do you solve it?':

Regularisation; the model has 'too many' features, and we should prioritise the more informative ones. L1 regularisation, in particular, allows for us to penalise solutions with many features and simplify the model (whilst retaining interpretability), so we could use a LASSO regression instead. L2 regularisation could also be used, but this doesn't, in general, reduce the number of features.

Unfortunately, I don't think these answers are textbook, so I would love some clarifications:

• Is this even a question about model fitting? Or is it really about variance-covariance matrices, portfolio risk, and/or CAPM-style financial management?
• An interviewer suggested using PCA instead of regularisation. I am not sure why that would be superior, since the principal components do not map to the original stocks you had in your portfolio)
• Does this apply to other models, which don't involve inverting $$X^\top X$$, or just linear regression?

As one of the interviewers suggested, the expected answer starts with PCA and SVD.

Before detailing it, let's take a paragraph about the way you seem to "misunderstand" the problem: suggesting LASSO or Ridge is out of scope. Indeed these techniques are based on the penalisation of a loss function and in this question: Where is the loss function you plan to penalise?
I would be the interviewer, such an answer would frighten me more than the candidate not proposing PCA.

Nevertheless, you get right the fact that this collinearity makes the inversion of $$X^T X$$ (in $$N^2$$) impossible because it is not full rank.
Not being full rank means that you have to operate in the orthogonal of its Kernel, and the way to identify the kernel is to diagonalise the matrix and to work in the orthogonal of its kernel.
What does it mean? You get the diagonal version of $$X^T X=P\Delta P^{-1}$$, and keep in mind that $$P^{-1}=P^T$$. Because the returns of $$K$$ stock are collinear, you should have $$K-1$$ zeros in the eigenvalues of $$\Delta$$, that are its diagonal values.
Look at the operation of multiplying $$X^T X$$ by a vector $$v$$ (whatever it is): $$X^T X\cdot v=P\Delta P^T \cdot v = P\cdot\big(\Delta(vP)^T\big).$$ To "work in the orthogonal of the kernel" means that when the work is "rotated" by $$P$$, the last $$k-1$$ components of your vector $$v$$ face zeros. They correspond to the last $$K-1$$ components of $$P$$.
This means that you can safely remove these coordinates: You can invert your matrix, if needed, in the space spanned by the $$N-K+1$$ first components of the PCA.

Numerically, since $$X$$ is in general rectangular with far more rows than columns, it is good to use a Singular Value Decomposition (SVD) decomposition. It prevents you from inverting a $$N$$ by $$N$$ matrix. It directly deals with the rectangular matrix.

In practice, it is not that easy because you find a lot of very small eigenvalues: are they zeros or not? is a complicated question. My advice is to get this Python code on scickit-learn, to keep only the first part and to try (last time I checked it succeeded to get returns of stocks from yahoo finance).
They are different approaches to deal with that: the first is to do some econometrics to identify the stocks that are collinear and to replace them with an "equivalent portfolio" (or just keep one of them), that is equivalent to position your problem in the orthogonal of the collinear returns. The second is to rely on Random Matrix Theory that will tell you how to "shrink" the eigenvalues of the $$X^T X$$ Matrix.

A last remark about your LASSO proposal, it is indeed far from stupid from a portfolio construction perspective. You ca have a look at Bruder, Benjamin, Nicolas Gaussel, Jean-Charles Richard, and Thierry Roncalli. "Regularization of portfolio allocation." Available at SSRN 2767358 (2013). It very clearly explains how most of the portfolio construction penalisations make sense. Nevertheless, it is not the answer that is expected first, because it opens the door to sophisticated questions about portfolio construction. Especially in an interview, but also in practice, you should start by setting a baseline model, before trying something more complicated.

• Thank you Charles for a very informative answer (and honoured for your help - I am a fan of your work, especially Managing Inventory Risk w/ Oliver G). Imagine X is full rank but contains many series with correlations > 0.9; we choose to do PCA but for now keep all 1000 stocks - so effectively we rotate the basis to the direction of maximum variance (I think this is effectively the same as the diagonalisation process you describe, with K=0). I don't have an intuition that the change of basis is as useful as reducing the number of features ... [1/2]
– Zac
May 24 at 11:43
• ... which could be achieved by regularisation (that is the loss function one could penalise; the number of features). I am not sure what you mean by 'work in the orthogonal of its kernel', but in this instance of full rank, it sounds like the last eigenvalues in the diagonal correspond to the eigenvectors of least variance, and so we might discard those. If this is correct, this is slightly hard to understand for me. Have I followed you correctly in the case that X is nondegenerate/full rank? [2/2]
– Zac
May 24 at 11:47
• Of course, we would normally use PCA for feature reduction since, in extremis, if we have 1000 stocks but 999 of them are almost-perfectly correlated with one another, there is no point 'training' a 1000-parameter model on the data (plus intercept) - we only really need the 2 principal components to model the data. But I was struggling to understand your suggestion that the change of basis is the most important part if the matrix is full rank, or whether the utility of PCA is more in feature reduction/selection which could also be achieved by LASSO for example.
– Zac
May 24 at 12:40
• @Lehalle: Thank you for the interesting answer. I follow some of it but I'm still a little confused. When building a covariance matrix, some of the "covariance" will be real so, if a PCA is performed, isn't that going to get rid of all of it ? For example, if it's part of a portfolio optimization, then throwing out the covariance, throws out important information. I must be not understanding something. May 24 at 14:24
• Zac: I'm not sure what works best for you but below is a list of references. Also, I'm pretty sure ( in retrospect ) that the answer is to discuss PCA-factor models as Lehalle suggested. link.springer.com/referenceworkentry/10.1007/… May 25 at 20:27