PCA(Principal Component Analysis) is the most interesting topic in QF. PCA is at the heart of quantitative data analysis. It is used in factor analysis, factor loadings, finding principal component of interest rate term structure for derivative and option pricing, data compression, eigenfaces( find the best match from a set of pictures with a , say, fuzzy face), you name it. So it behooves one to understand the basics because it will help one to understand the more advance concepts in PCA calculations.
I hope the answer and the thinking behind the logic is appealing and matches yours.
There are many ways to do this. I am using SVD, Eigen decomposition, which will work with matrix A being positive semi-definite. RMT is a relatively new field. To cover the basics this is a good start. Normally eigenvectors and eigenvalues relationship is shown as
$$ A v = \lambda v $$
where $ v \space is \space eigenvector$ and $\lambda \space is \space eigenvalue $
An interesting property of eigenvector is that is you multiply it with a scalar, and multiply with A on the left, you will find vector reproduce on the right side, with factor k, with same eigenvalues. ( Exercise).
Mind you, it is a mistake to think that eigenvectors are same as principal components, however, they are related as shown below.
If you have a random dataset X as below, then what is principal component and how is data related to A matrix?
Numerical example shows using MATLAB.
X = [269.8 38.9 50.5
272.4 39.5 50.0
270.0 38.9 50.5
272.0 39.3 50.2
269.8 38.9 50.5
269.8 38.9 50.5
268.2 38.6 50.2
268.2 38.6 50.8
267.0 38.2 51.1
267.8 38.4 51.0
273.6 39.6 50.0
271.2 39.1 50.4
269.8 38.9 50.5
270.0 38.9 50.5
270.0 38.9 50.5
];
[n m]=size(A)
n =
15
m =
3
Let us convert X to zscores, we can B=zscore(X) or for clarity
B = (X - repmat(mean(X),[n 1])) ./ repmat(std(X),[n 1])
B =
-0.0971 -0.0178 0.0636
1.3591 1.5820 -1.5266
0.0149 -0.0178 0.0636
1.1351 1.0487 -0.8905
-0.0971 -0.0178 0.0636
-0.0971 -0.0178 0.0636
-0.9932 -0.8177 -0.8905
-0.9932 -0.8177 1.0178
-1.6653 -1.8842 1.9719
-1.2173 -1.3509 1.6539
2.0312 1.8486 -1.5266
0.6870 0.5155 -0.2544
-0.0971 -0.0178 0.0636
0.0149 -0.0178 0.0636
0.0149 -0.0178 0.0636
> cov(B)
ans =
1.0000 0.9902 -0.8433
0.9902 1.0000 -0.8831
-0.8433 -0.8831 1.0000
> corr(B)
ans =
1.0000 0.9902 -0.8433
0.9902 1.0000 -0.8831
-0.8433 -0.8831 1.0000
> corr(X)
ans =
1.0000 0.9902 -0.8433
0.9902 1.0000 -0.8831
-0.8433 -0.8831 1.0000
Interesting Note: for standardized data mean=0, variance =1,
$$A=cov(B)=corr(B)=corr(X)$$
However, note
$$ cov(B) <> cov(X) $$
Therefore be careful of how you calculate the covariance matrix.
Note here A used in beginning is the covariance matrix of standardized input.
Decomposing matrix A or cov(B) gives us eigenvectors V and eigenvalues D as below.
[V D]=eig(cov(B))
V =
0.6505 0.4874 -0.5825
-0.7507 0.2963 -0.5904
-0.1152 0.8213 0.5587
D =
0.0066 0 0
0 0.1809 0
0 0 2.8125
Interesting to note that B'*B gives you a matrix proportional to cov(B).
B'*B
ans =
14.0000 13.8622 -11.8068
13.8622 14.0000 -12.3640
-11.8068 -12.3640 14.0000
It is off by a factor of 14.
B'*B/14
ans =
1.0000 0.9902 -0.8433
0.9902 1.0000 -0.8831
-0.8433 -0.8831 1.0000
Dividing by 14 gives you the covariance matrix , A or cov(B).
Why is this so. Because Covariance estimator is given as:
$$ \frac{1}{n-1} \sum_{i=1}^{n} (X_i- \overline{X}) {(X_i- \overline{X})}'$$
where $ \overline{X}=0 $
we get $$ Covariance matrix (estimator)=\frac{1}{n-1} B B' $$
That mystery is solved.
Note we can get matrix A or cov(B), also using Spectral Decomposition
V*D*V'
ans =
1.0000 0.9902 -0.8433
0.9902 1.0000 -0.8831
-0.8433 -0.8831 1.0000
Principal components are defined as scores in Matlab. We can get the principal components from eigenvalue
> score=B*V
score =
-0.0571 -0.0003 0.1026
-0.1277 -0.1226 -2.5786
0.0157 0.0543 0.0373
0.0536 0.1326 -1.7779
-0.0571 -0.0003 0.1026
-0.0571 -0.0003 0.1026
0.0704 -1.4579 0.5637
-0.1495 0.1095 1.6299
0.1041 0.2496 3.1841
0.0319 0.3647 2.4306
0.1093 0.2840 -3.1275
0.0892 0.2787 -0.8467
-0.0571 -0.0003 0.1026
0.0157 0.0543 0.0373
0.0157 0.0543 0.0373
We have three principal components above. Let us name them from left to right as PC1, PC2 and PC3. If you look at the eigenvectors last column you will see
[-0.5825 -0.5904 0.5587]'
The way the score is obtained from matrix multiplication between B and V, you can see that PC3 consists of -58.25% of first factor (in B) , -59.04% of 2nd factor and 55.87% of 3rd factor.
Which one is the most dominant?
> var(score)
ans =
0.0066 0.1809 2.8125
The right most vector, PC3, explains the most of the variance in the data X.
you can do
varB=sort(var(score),'descend')
varB =
2.8125 0.1809 0.0066
cumsum(varB) / sum(varB)
ans =
0.9375 0.9978 1.0000
So the dominant principal component(PC3) explains 94% of the variance. If you are finding the principal component that explains the variance in data then PCA works fine. If you are using PCA for data that has other information content then you need something different. Also, covariance matrix can be skewed by the outliers, I find that interesting. What if I am only interested in outliers. Outliers could be representing the Black Swans?! Then may be you are interested in the least important PC. You can plot the scores and see how it matches the outlier data.
One last thing, how do you get back the original data from score or principal components?
As I showed earlier
> B = (X - repmat(mean(X),[n 1])) ./ repmat(std(X),[n 1])
so
> X = (B .* repmat(std(X),[n 1]) + repmat(mean(X),[n 1]))
Which really is how we get X from B. How about the score?
We know score is B*V so B=score*V' because then B=B*V*V' and V*V'=1 or identity matrix.
therefore,
> X = (score*V' .* repmat(std(X),[n 1]) + repmat(mean(X),[n 1]))
In pseudo equation form, PC: principal component
$$ original \space data = PC * eigenvector' * stdev(X) + mean $$
I have answered the following question in this
- how to get eigenvectors, eigenvalues and PCA
- how to relate eigenvectors to PCA components
- determine which eigenvalues explain the most variance
- how much do the factors contribute to the principal component
- double check which principal component is dominant
- How to get the original data back
