3
$\begingroup$

I used ten year daily data for 407 stocks and computed the daily and monthly covariance matrices. Since I have more variables than observations for the monthly matrix, I wasn't surprised to find the matrix to be not invertible (and hence useless for portfolio optimization). I was surprised to see the daily covariance matrix not invertible. I then tried to shrink the matrix with the Ledoit-Wolf shrinkage estimator using the package tawny. It didn't help. It makes the covariance matrix really, really small, but no invertible.

Does anyone have any suggestions what could be the problem? How could I improve the covariance matrix?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

Are your 407 stocks all different? No A and B listings contained that are strongly if not perfectly correlated? The observation that the daily covariance matrix is singular makes me wonder.

You can try the package corpcor for another shrinkage estimator.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I reduced the sample to 50 stocks and now the matrix is invertible for daily and monthly returns. I used the Ledoit,Wolf shrinkage estimator for the covariance matrix only and now I get really small portfolio variances, implying unrealistic Sharpe-ratio. I assume this has to do with the fact that I didn't shrink the return vector. Which shrinkage estimator would you use on the return vector? $\endgroup$
    – User1111
    Commented Aug 28, 2015 at 14:22
  • $\begingroup$ If you get small portfolio variances it might be because you have long and short positions in similar stocks. You might consider a no-short sales, i.e. non-negative portfolio weights constraint. $\endgroup$
    – Alex C
    Commented Aug 29, 2015 at 19:07
  • $\begingroup$ Thanks Alex C, that could be a problem, I'll check that out! $\endgroup$
    – User1111
    Commented Aug 30, 2015 at 20:07
  • $\begingroup$ @User1111 No, I don't think that shrinking the return vector will do any good. What do you mean by small variances? For sharpe you divide mean by the square-root of variance ... this is bigger than variance. Do you use proper scaling (for annualizing a monthly Sharpe-Ratio you multiply by $\sqrt{12}$, \sqrt{250} for daily data. Why don't you get a non-singular covariance matrix with the original data? Do you have constant time series or time series with missing values that are replaced by constants? $\endgroup$
    – Richi Wa
    Commented Aug 31, 2015 at 6:27
3
$\begingroup$

The problem with Ledoit-Wolf is that it's very sensitive to outliers.

You should try these:

  1. DCC GARCH
    • unfortunately, not available in Python
  2. Exponentially weighed moving average (EWMA)
    • gives slighly worse results than DCC-GARCH
  3. Minimum Covariance Determinant
    • suggestted by Scikit-Learn
  4. bootstrap
    • could be used to calculate confidence interval, then you can decided how conservative you want to be

Here are some good references:

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ It'll be great if you could elaborate other methods a bit more $\endgroup$
    – Xiaoxiong Lin
    Commented Dec 11, 2018 at 15:32
3
$\begingroup$

In theory, the Ledoit and Wolf shrinkage estimator is supposed to guarantee a positive-definite matrix, given that it adds a positive-definite matrix (the target) to a semi-positive one (the sample covariance).

I can see four reasons why you didn't get a positive-definite matrix:

  1. Your true covariance is effectively not full rank, i..e you have perfect multicolinearity

  2. Your target is not positive definite? That is something you can check easily. Taking (a multiple of) the identity will however guarantee it is

  3. Your sample covariance matrix is not semi-positive definite: this can happen when you had missing values, and used a cor(x, "pairwise.complete.obs") approach

  4. There are bugs in the code (tawny has indeed bugs in the Ledoit Wolf estimator, as of version 2.1.7). Check alternatives like nlshrink::linshrink_cov() and CovTools::CovEst.2003LW()

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.