2
$\begingroup$

I am trying to optimize a simple portfolio using several random weights and choosing the best. When the number of assets is large I get a covariance matrix with 'nan' values because some asset pairs do not have trading days in common.

How should I treat the 'nan' values?

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Sounds like the 'covariance matrix' is not positive definite, i.e. not a covariance matrix. Where did cov_matrix come from? $\endgroup$
    – nbbo2
    May 11, 2017 at 16:30
  • $\begingroup$ Do you get NaN values for your portfolio weights? Or during the computation of the covariance matrix? I'm confused. I don't see any NaNs in the portfolio weights vector $\mathbf{w}$. Are you saying that $\mathbf{w}' \Sigma \mathbf{w}$ gives you a NaN but that $\Sigma$ doesn't have a NaN and $\mathbf{w}$ doesn't have a NaN? $\endgroup$
    – Matthew Gunn
    May 11, 2017 at 16:52
  • $\begingroup$ @MatthewGunn Yes, w'Σw gives me a NaN but Σ doesn't have a NaN and w doesn't have a NaN? And it only happens as the number of assets increases. $\endgroup$
    – Pedro Rio
    May 11, 2017 at 17:14
  • $\begingroup$ @noob2 I had several panda time series of asset returns and computed with cov_matrix = returns.cov() $\endgroup$
    – Pedro Rio
    May 11, 2017 at 17:17
  • $\begingroup$ @PedroRio Is there an inf anywhere? Absurdly large or small numbers? I'm struggling to see you can get a nan from matrix multiplication except through something like (-np.inf) + np.inf. Check w.min(), w.max(). If you don't take the square root, what is the value you get for the portfolio variance? If your covariance matrix is rank deficient and not quite positive semi-definite, perhaps $\mathbf{w}' \Sigma \mathbf{w}$ is somehow negative? Those are some ideas but you're going to have to track down what is going wrong. $\endgroup$
    – Matthew Gunn
    May 11, 2017 at 18:23

2 Answers 2

Reset to default
1
$\begingroup$
  • Your estimated covariance matrix includes nan entries.
  • The current Pandas.cov function already makes a best effort to estimate covariance based upon available data by ignoring nan/null values.

This implies that to obtain a nan in the estimate of covariance, you must have at least two return series that have ZERO time periods in common!

Your question is ill posed

  • What's the correlation between returns of the Dutch East India Company (1602-1799) and Google (2004 - now)? It's an unanswerable and non-sensical question.
  • And if your portfolio optimizer says to put $\frac{1}{2}$ your portfolio in Dutch East India Company and $\frac{1}{2}$ in Google, how are you going to do that?

A direction to move in

If you're going to work with securities that enter and leave your sample, you need to do something more sophisticated than estimate some unconditional covariance matrix with sigma = mydata.cov() and using that to choose portfolio weights.

  • If the point is come up with portfolio weights for time $t$, it doesn't make sense to include securities which one cannot invest in at time $t$!
  • You need some notion of $\Sigma_t$, an estimate that's designed for time $t$.

And replacing nan with 0 is not a sensible thing to do! The average covariance term is not zero. Systematic aggregate risk exists and this manifests itself in greater than zero covariance terms.

Improve this answer
$\endgroup$
4
$\begingroup$

This is a common problem in covariance matrix estimation, with several possible solutions. One of the simplest involves two steps:

(1) You compute each element of the covariance matrix on a 'best efforts' basis, meaning you take the covariance of the two time series involved after REMOVING any data pairs having a N/A value. (Note that this means each element of the matrix will be based on a different number of observations, which means the resulting matrix is not a standard covariance matrix, it may not be positive definite for example). I assume that for any two time series there are at least a few common observations, otherwise it is an ill-posed problem as Matthew Gunn pointed out.

(2) You "massage" the resulting matrix to make it positive definite (and thus acceptable for use as a covariance matrix) using the routine nearPD which is available in R link

[Even after all this work, a large covariance matrix will be very 'noisy' and of poor quality. You should consider further steps such as 'shrinkage' link before you use the results to find an optimum portfolio].

Improve this answer
$\endgroup$
9
  • $\begingroup$ nearPD is nice functionality $\endgroup$
    – FinanceGuyThatCantCode
    May 12, 2017 at 18:51
  • 1
    $\begingroup$ The current version of Pandas.cov already calculates covariance "excluding NA/null values" which is the "best efforts" basis you are referring to. $\endgroup$
    – Matthew Gunn
    May 12, 2017 at 19:20
  • $\begingroup$ An example of nearPd I did last year: stackoverflow.com/questions/36153022/… $\endgroup$
    – rbm
    May 12, 2017 at 19:48
  • $\begingroup$ Most "nearPD" implementations just do it via evd and then removing negative entries. There are some better ways (Imo) based on vectors in some hypersphere. $\endgroup$
    – will
    May 12, 2017 at 19:51
  • 2
    $\begingroup$ While I appreciate your concern for negative eigenvalues and the possibility of portfolio weights that purport to achieve negative volatility, the OP almost certainly has a far more basic problem of having delisted securities in his return matrix! $\endgroup$
    – Matthew Gunn
    May 12, 2017 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.