If you look at changes of the points on the yield curve, then you probably find something stationary - right? Applying PCA on the covariance of these changes makes sense.
E.g. you will find out that on PC describes a parallel shift (a change in the yield curve). Look at this question too: What do eigenvalues/eigenvectors of the yield/forward rates covariance matrices mean?
