Given a time series of a set of N assets (let's say 100), how can I find the optimal portfolio, with the constraint that only n<N assets (let's say 10) can be in the portfolio? With 'optimal portfolio' I mean the 'efficient portfolio' (so minimum variance for a given return, or maximum return for a given variance). Within the modern portfolio theory framework one can find the weights for all the N assets, but what if I want n<N assets only?
This is generally considered a hard problem to solve because it is an example of a Mixed Integer Non-Linear Programming problem, and would typically require access to a commercial optimisation routine to solve. The problem requires the addition of N variables ($v_i, i=1,N$) to the general Markowitz optimization problem. $v_i$ can take the value 0 or 1 depending on whether an asset is included in the final portfolio. Each asset has modified variable bounds $= l_iv_i < x_i < u_iv_i$, and we have a final constraint on the number of total variables $\Sigma_i v_i <= n$.
Mathworks shows some of the theoretical background at https://uk.mathworks.com/help/optim/ug/mixed-integer-quadratic-programming-portfolio-optimization-solver-based.html. MATLAB's solution requires multiple calls to a Mixed Integer Linear Programming routine.
People have also solved this using more ad-hoc methods in the past (genetic algorithms, simple heuristics). So, for example, one can optimise with the full N assets, remove the smallest position, and repeat until there are only n assets left. This approach is not guaranteed to be optimal but may be practically workable. In particular, when you consider that there is significant estimation error on the tracking error, any further improvement to a truly optimal solution may not be worthwhile.