5 added 17 characters in body edited Jan 11 '12 at 23:22 Ryogi 2,8411515 silver badges2727 bronze badges This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) remain part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$$$correlation \;\; matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. To recap, all RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (1999, 2000) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations in what new eigenvalues you assign to the noisy eigenvalues (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing so you cannot set these to zero. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue). Also, one of the most accessible introductions to Random Matrix theory are the papers by V. Plerou et al. The Capital Fund Management Team - Bouchaud, Pafka, Potters, et al has more frontier research on RMT. This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) remain part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. To recap, all RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (1999, 2000) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations in what new eigenvalues you assign to the noisy eigenvalues (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing so you cannot set these to zero. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue). Also, one of the most accessible introductions to Random Matrix theory are the papers by V. Plerou et al. The Capital Fund Management Team - Bouchaud, Pafka, Potters, et al has more frontier research on RMT. This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) remain part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$correlation \;\; matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. To recap, all RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (1999, 2000) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations in what new eigenvalues you assign to the noisy eigenvalues (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing so you cannot set these to zero. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue). Also, one of the most accessible introductions to Random Matrix theory are the papers by V. Plerou et al. The Capital Fund Management Team - Bouchaud, Pafka, Potters, et al has more frontier research on RMT. 4 added 4 characters in body edited Jan 10 '12 at 5:19 Ram Ahluwalia 10.5k11 gold badge4545 silver badges8585 bronze badges This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) remain part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$covariance Matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$$$matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. To recap, all RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (1999, 2000) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations in what new eigenvalues you assign to the noisy eigenvalues (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing so you cannot set these to zero. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue). Also, one of the most accessible  introductions to Random Matrix theory are the papers by V. Plerou et al.  The Capital Fund Management Team - Bouchaud, Pafka, Potters, et al has more frontier research on RMT. This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) remain part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$covariance Matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. To recap, all RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (1999, 2000) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations in what new eigenvalues you assign to the noisy eigenvalues (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing so you cannot set these to zero. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue). Also, one of the most accessibleintroductions to Random Matrix theory are the papers by V. Plerou et al. The Capital Fund Management Team - Bouchaud, Pafka, Potters, et al has more frontier research on RMT. This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) remain part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. To recap, all RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (1999, 2000) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations in what new eigenvalues you assign to the noisy eigenvalues (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing so you cannot set these to zero. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue). Also, one of the most accessible  introductions to Random Matrix theory are the papers by V. Plerou et al.  The Capital Fund Management Team - Bouchaud, Pafka, Potters, et al has more frontier research on RMT. 3 added 464 characters in body edited Jan 10 '12 at 4:46 Ram Ahluwalia 10.5k11 gold badge4545 silver badges8585 bronze badges This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) are stillremain part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$covariance Matrix = eigenVectors * diag( eigenvalues ) * t(eigenvectors)$$$$covariance Matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. AllTo recap, all RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (19991999, 20012000) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations herein what new eigenvalues you assign to the noisy eigenvalues (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing so you cannot set these to zero. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue). Also, one of the most accessibleintroductions to Random Matrix theory are the papers by V. Plerou et al. The Capital Fund Management Team - Bouchaud, Pafka, Potters, et al has more frontier research on RMT. This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) are still part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$covariance Matrix = eigenVectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. All RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (1999, 2001) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations here (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue) This is correct: "The general idea of cleansing a correlation matrix via random matrix theory is to compare its eigenvalues to that of a random one to see which parts of it are beyond normal randomness." This is not correct: "These are then filtered out and one is left with the non-random parts." The term "filtering", although used extensively in the literature, is misleading because the eigenvectors from the original correlation (or covariance matrix) remain part of the matrix and the sum of the eigenvalues does not change. In all RMT filtering procedures, the matrix is decomposed and re-built via the eigenvector decomposition theorem: $$covariance Matrix = eigenvectors * diag( eigenvalues ) * t(eigenvectors)$$ Notation: t() is the transpose operator, * is matrix multiplication, and diag() is an operator that creates a diagonal matrix In RMT we only perform operations on the diagonal matrix of eigenvalues resulting from the decomposed matrix. The eigenvectors are left untouched. Also, the sum of the eigenvalues is preserved pre-and post cleansing. Since the sum of the eigenvalues also equals the trace of the covariance matrix, this ensures that the variance of the system is preserved. To recap, all RMT procedures follow this four-step process: The first step is, as you point out, identifying the upper noise band of eigenvalues predicted by a de-meaned random matrix of the same sigma using the Marcenko-Pastur law. Matrices that are exponentially weighted use a Power-law. The matrix is decomposed via the eigenvector decomposition theorem The diagonal of eigenvalues is cleansed. For example, the method of Laloux (1999, 2000) is to assign the eigenvalues for all "noisy" eigenvalues beneath the upper-noise band to the average of all such noisy eigenvalues. There are several variations in what new eigenvalues you assign to the noisy eigenvalues (power-law method, Krzanowski, etc.). Regardless - in all RMT methods the sum of all the eigenvalues is preserved post-cleansing so you cannot set these to zero. Now we re-build a "cleansed" or "filtered" covariance matrix using the same eigenvectors and the revised eigenvalue via the eigenvector decomposition theorem (in reverse this time with the same eigenvectors). Digression: You could come up with your own techniques so long as the constraints above are respected. For example, since we know that the principal/top eigenvectors have significant non-normal structure or skew, you could "filter" those eigenvalues that correspond to eigenvectors that exhibit normal structure. Also, for what it's worth, in my experience filtering methods that have a successively increasing distance from one eigenvalue value to the next perform better (i.e. for example a recursive definition such as the value of the 2nd smallest eigenvalue is twice the value of the smallest eigenvalue, the value of the 3rd smallest eigenvalues is twice the value of the 2nd smallest eigenvalue, etc. up to the last noisy eigenvalue). Also, one of the most accessibleintroductions to Random Matrix theory are the papers by V. Plerou et al. The Capital Fund Management Team - Bouchaud, Pafka, Potters, et al has more frontier research on RMT. 2 added 283 characters in body edited Jan 10 '12 at 4:34 Ram Ahluwalia 10.5k11 gold badge4545 silver badges8585 bronze badges 1 answered Jan 10 '12 at 4:28 Ram Ahluwalia 10.5k11 gold badge4545 silver badges8585 bronze badges