I am a first year university student.

I am trying to replicate an Index, for instance SP500. But instead of doing a full replication (by buying all the stocks), I wonder : How can I choose a portfolio of N% of index stocks to minimize the portfolio volatility

I was thining I could do this by using covariance matrix and do a optimization algorithm using Python.

Do you think this is the best solution ?

Thank you very much for your time,


  • $\begingroup$ So you want to track the index for accuracy (replication) but not track the index so as to have a lower volatility? I think you need to choose which you want to do. Another alternative is to formulate your question more like "How can I replicate the index with up to X% tracking error while minimizing the volatility?" or "How can I choose a portfolio of N% of index stocks to minimize the portfolio volatility?" $\endgroup$ – kurtosis Aug 14 '20 at 7:56
  • $\begingroup$ Hi Kurtosis, thank you for your help. I edited my post. I was refering to the second option indeed : "How can I choose a portfolio of N% of index stocks to minimize the portfolio volatility $\endgroup$ – Alexis Aug 14 '20 at 8:43
  • $\begingroup$ An interesting problem. The optimization may be non-trivial to do, as there are no easy ways (that I know of) for constraining the number of stocks in the portfolio to be exactly the required number. (You might have to start with N=1 (pick the least volatile stock), and then find the best stock to add to form the N=2 portfolio and so on for increasing N). $\endgroup$ – noob2 Aug 14 '20 at 14:12
  • 3
    $\begingroup$ Minimizing volatility while limiting number of names isn't really replicating an index, it's a separate exercise. Your objective should be to minimize TE (ie, sub-portfolio moves with SP500) with N < M names. $\endgroup$ – Chris Aug 14 '20 at 17:07
  • $\begingroup$ @Chris I agree, but this would not be the first (or even 100th) time a student has used the wrong terminology to describe what they want to do. I also agree it isn't the problem I would choose to solve, but that is a subjective opinion. $\endgroup$ – kurtosis Aug 14 '20 at 17:44

I agree with the comments that this is a weird objective, and minimizing tracking error makes more sense. However, this clarified objective can be solved.

What you have looks like a mixed integer program optimization problem since we effectively have intermediate variables which multiply the weights by 0 or 1. What you want could even be a multi-objective MIP, if you subsequently maximize return holding variance constant; however, that would be more complicated and you already have enough work ahead of you.

Your single objective MIP would look like so: $$ \begin{align} \min &~w^T\Sigma w \\ \text{s.t.} &~||w||_0\leq N^* && \text{(constrain non-zero weights)} \\ &~||w||_1=1. && \text{(weights sum to 1)} \end{align} $$ where $N^*$ is a number instead of a percentage.

Also, you should realize that IP and MIP optimizations use heuristics since they are NP-hard problems to solve. Therefore, you might not be able to ensure optimality, solving could take more time than you expect, and the solution might not be very robust (i.e. changing inputs or constraints might lead to a very different solution).

You might do well to consult or.SE (I know, it can be sleepy... but they are the experts on optimization) as well as reading this post from math.SE.


Further to comments above, wondering if this isn't a problem intuitively solved by LASSO regression?


The objective being to minimise the objective function:

enter image description here

Essentially you add in a penalty term (lambda above) for taking up regression weights. So Y would be your series of S&P returns, the X's would be your stock returns and the Betas would be your stock weights. At lambda = 0, this would be an unconstrained OLS regression, in which the betas would be equally unconstrained stock weights in the index.

But as you increase lambda, the cost function (latter half) of allocating weights to stocks increases. So the model will start to tolerate some tracking error (essentially the first half of the objective) to reduce the cost function. The model will start to shrink the most redundant variables (ie the weight of the least important stocks in this case) down to zero. So as lambda increases, your N number (the count of stocks with a beta >0) will start to shrink.

Two caveats spring to mind:

  • the model could settle on a non-negative weight for a stock, ie the optimal replication would be to short-sell that stock! So you'd probably want to impose some non-negativity constraint on your betas.

  • the model could be "high variance, low bias". Suppose you wanted to replicate the S&P500 with only 50 stocks. So you take, say, all the last 6m returns for the index and its constituents. That's only 130 days for 500 stocks, ie dimensions outnumber observations. The LASSO model comes up with its "optimal 50". But if you then go back a month later and re-run the exercise, you might find that the 50 then contained a lot of different stocks to the 50 you got a month before. The tracking error might not be very different at all between the two sets of stocks. Both might thus be "good" replications of the index - just the set that is "best" at any point in time might change quite frequently over time.

Hope this helps.


Happy to help; but there is one simple question...

Are you trying to "replicate" the S&P with a single index weight for each stock from which the index cannot deviate? Or are you trying to "replace" the S&P with a set of constituents that have the lowest volatility/variance? Or are you trying to "reproduce" the S&P with some kind of fixed weights that somehow doesn't develop into a growing tracking-error problem? Maybe, you weights are moving averages of index weights, so don't fluctuate so much?

What are you trying to achieve here? These are NOT the same thing at all.

If you're asking "what's the best proxy for the index using only N stocks?", that an incredibly complicated Machine Learning task :-) To do that, you'd have to build a model that was always willing to ignore one stock, in favour of using another as its proxy, with each iteration. And iterate, thousands of times.


stratified sampling? probably want to avoid all kinds of tiny holdings and unnecessary complexity while maintaining a maximum tracking error constraint. for something like a high yield index, tracking with the exact ultimate underlying assets may be impossible i.e. hundreds of HY issues that trade sporadically and may not be divisible ...... NOTE: this may be really obvious.

i am aware of someone at a huge pension fund who tried to replicate index performance on less liquid indexes on a perfect dollar-for-dollar basis........... got insanely taken advantage of by dealers for discretionary (correct word?) index changes because they knew exactly what/when he'd be making changes. and he was one of the few people actually making these changes. basically i mean additions/deletion. not sure what else.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.