In portfolio management one often has to solve problems of the quadratic form $$ w^T \Sigma w + w^T c \rightarrow \min_{\omega} $$ with portfolio weights $w \in \mathbb{R}^N$ a constant $c \in \mathbb{R}^N$ and a covariance matrix $\Sigma \in \mathbb{R}^{N \times N}$. Furthermore we assume real world continuous and binary (e.g. cardinality) constraints.
For estimating the covariance matrix $\Sigma$ we can e.g. use the sample covariance of the returns of all assets - let's call this an asset-by-asset model. It is known that some care has to be taken here if $N$ is big and so forth, but this is not the point of this question.
On the other hand we can define factors $(F_k)_{k=1}^K$ with $K<N$. These factors have a covariance matrix $\Sigma_F \in \mathbb{R}^{K \times K}$. Denoting by $r_i$ the return of asset $i$ with $i = 1, \ldots, N$, we can write: $$ r_i = \sum_{k=1}^K e_{i,k} F_k + \epsilon_i, $$ with the meaning that the variation of the return is described by the exposures $e_{i,k}$ to the factors and some purely idiosyncratic risk $\epsilon_i$. In this case the covariance matrix of returns is given by $$ \hat{\Sigma} = e \Sigma_F e^T + \mathop{diag}(Var[\epsilon_1],\ldots,Var[\epsilon_N]), $$ where $e \in \mathbb{R}^{N \times K}$ is the matrix of all exposures and the "diag" part adds the idiosyncratic parts of variance at the main diagonal. Note that $\hat{\Sigma} \in \mathbb{R}^{N \times N}$.
I sometimes read that a problem with covariance matrix $\hat{\Sigma}$ from the factor model is easier to solve than with $\Sigma$. Is this true? If yes, then how can we see this? My personal answer so far is: no. Because both are $N \times N$ matrices and the structure does not help in general.
Especially if the factors are not orthogonal - what do we gain? If the factors come from a PCA then we might gain something but I wonder how many PCs we would need e.g. in a global equities portfolio ...