If you look at changes of the points on the yield curve, then you probably find something stationary - right? Applying PCA on the covariance of these changes makes sense.

E.g. you will find out that on PC describes a parallel shift (a change in the yield curve). Look at this question too: https://quant.stackexchange.com/questions/7202/what-do-eigenvalues-eigenvectors-of-the-yield-forward-rates-covariance-matrices/7204#7204