Imagine you have a matrix of returns (n assets, t days) and want to compute c statistical risk factors using PCA/SVD, so that you get (n, c) matrix of factor loadings and (c, t) matrix of factor returns.
Is it better to decompose the matrix of returns directly with SVD, or to subtract column means first, as PCA does in scikit-learn? Why?
Is it better ...
It depends on your use case. If you want to spot outliers, SVD against the raw data may be preferable.
The column mean is an equal weighted "market return" estimate. I find it preferable to standardize the rows (individual asset return series).
To reduce the influence of volatile assets, standardize the individual asset returns $x_{i,\tau}$ $$ \begin{align} \hat{\mu_i} &= \frac{1}{t}\sum\limits_{\tau=1}^t x_{i,\tau} &\\ ~\\ \hat{\sigma_i} &= \frac{1}{t-1}\sum\limits_{\tau=1}^t \left(x_{i,\tau}-\hat{\mu_i}\right)^2 \\ ~\\ z_{i,\tau} &= \frac{x_{i,\tau} - \hat{\mu_i}}{\hat{\sigma_i}} \end{align} $$
$Z Z^\textrm{T}$ is the asset correlation matrix.
Decomposing $Z^\textrm{T} Z$ with PCA will yield characteristic return series. See Connor and Korajczyk (1986), Performance measurement with the arbitrage pricing theory: A new framework for analysis. You would then regress your asset returns on these characteristic time series to obtain factor loadings.
Alternatively, if you decompose $Z$ with SVD, you will directly obtain characteristic portfolios (factor mimicking portfolios) and characteristic time series (factor returns) for a rank-$c$ approximation of the observed returns. $$ \begin{align} Z &= U S V^\textrm{T} \\ & ~ \\ U &= \begin{bmatrix} ~\\ u_1 \ldots u_c \\ ~ \end{bmatrix} \quad \text{characteristic portfolios,}\\ ~ \\ V S &= \begin{bmatrix} ~ \\ v_1 \ldots v_c \\ ~\end{bmatrix} \begin{bmatrix}s_{1,1} & ~ & ~ \\ ~ & \ddots & ~ \\ ~ & ~ & s_{c,c} \end{bmatrix} \quad \text{characteristic time series.}\end{align} $$
Matrix dimensions - $ \quad Z_{n \times t}\quad U_{n \times c}\quad V_{t \times c} \quad S_{c \times c} $
The original returns from standardized returns: $$ \begin{align} x_{i} &= \hat{\sigma_i} z_{i} + \hat{\mu_i} \\ ~\\ &\approx \hat{\sigma_i} u_i S V^\textrm{T} + \hat{\mu_i}~. \end{align} $$
To match total model variance for asset $i$ to observed $\hat{\sigma_i}^2$, add idiosyncratic variance of $$ \begin{align} \psi_i^2 &= \hat{\sigma_i}^2 \left( 1 - u_i S^2 u_i^\textrm{T} \right) \end{align} $$ where idiosyncratic variance equals total variance minus common factor variance.
