Suppose we have $A$ a $T \times N$ matrix of daily returns for an asset universe of $N$ items, $b$ a $(T,)$ vector of daily returns for a target asset, $x$ a $(N,)$ weighting vector. We want a portfolio that replicates daily returns $b$ as closely as possible, subject to the constraint that portfolio weights $x$ sum to 1 and $x$ is non-negative (no shorting). One might use this for tax loss harvesting or hedging.
If we first consider the unconstrained case:
$$ \min_x \sum_t^{T} (A \cdot x - b) $$
This is nothing more than least-squares approximation / regression, whose objective can be re-written as:
$$ |Ax-b|^2 = (Ax-b)^T(Ax-b) = x^T{A^T}Ax - 2x^TA^Tb + b^2 $$
Taking the derivative wrt $x$ and setting to 0,
$$ 2A^TAx - 2A^Tb = 0 \\ x = (A^TA)^{-1}A^Tb $$
To handle equality constraints such as $x$ summing to 1, and inequality constraints like $x$ being non-negative, one can set up a constrained least squares problem, solve the KKT matrix, and still derive a closed form solution, albeit one that has cubic complexity due to necessity of inverting matrices.
However, the portfolio manager may have a few other constraints - maybe they prefer to keep $x$ relatively sparse, so they can just buy about 3-4 assets rather than allocating a small percentage across thousands of assets. There are more complex techniques like ridge regression, lasso regression, and so on, but these start to involve stepwise optimization (e.g. coordinate descent).
My question is one about how the portfolio optimization industry actually optimizes in practice: is it more similar to analytic convex optimization with simple inequality / equality constraints, and maybe a bit of hacks on top of the Least-square regression (for instance, rounding off small values of $x$), or is more step-wise optimization the norm? Given a sufficiently complex objective, would it make sense just to switch to a more black-box optimization algorithm, like a genetic algorithm or some kind of stochastic gradient descent optimizer?