Regularizers to compute Minimum Variance Portfolio weights

Question

I need to compute the mimimum variance portfolio using different regularizers, to compare the results and use validation methods to find the optimal parameters. Currently my work has been performed using python. Following question displays my current work

Efficient frontier doesn't look good

What I need to do is, I need to extend this work to be used with different regularizers. The dataset is 48_Industry_Portfolios_daily dataset. There are enough tutorials to uses lasso and Ridge, when I'm having a dataset with class variables. But not for such a situation.

Can someone kindly share with me the exact steps I need to follow, to extend my code, so I can use regularizers to compare portfolio variances. Even an overview to do this is highly appreciated.

I understand the following calculations,

https://towardsdatascience.com/ridge-and-lasso-regression-a-complete-guide-with-python-scikit-learn-e20e34bcbf0b

But, I'm not sure how these calculations can be used for portfolio variance comparison

Answer 1

The Minimum Variance Portfolio (without constraints, other than the weights sum to one) is usually found as $$w=\frac{\Sigma^{-1} \iota}{\iota^T\Sigma^{-1} \iota}$$ where $\Sigma$ is the Covariance matrix and $\iota$ is a vector of all ones.

However, there is another (equivalent) way to find it. Memmel and Kempf (2006) SSRN 940367 showed that you can find it from the returns time series using an OLS regression:

You select one (for example the n-th) time series of returns and you regress it on the differences with the other n-1 time series (see equation (5) in their paper):

$$\tilde{r}_n= \alpha +\beta_1 (\tilde{r}_n-\tilde{r}_1) + \beta_2 (\tilde{r}_n-\tilde{r}_2) +\cdots + \beta_{n-1}(\tilde{r}_n-\tilde{r}_{n-1})+\tilde{\epsilon}$$

The coefficients $\beta_1,\cdots,\beta_{n-1}$ thus found are in fact equal to the desired minimum variance portfolio weights $w_1,\cdots,w_{n-1}$. (Strange but true). The last weight $w_n$ can be found by $w_n=1-\sum_{i=1}^{n-1} w_i $.

Knowing this, how can we find the minimum variance portfolio using regularizers? In the same way, except that instead of using an OLS regression routine, we use a routine that performs regularized regression, for example Ridge Regression.

Therefore by using any available code that performs Ridge Regression or Lasso we can find a regularized Min Variance Portfolio. There is no need to write any new code, we just have to call the routine with the right parameters.

(As mentioned in another question Compare portfolio variance using different regularizers the variance thus found will be larger than the true minimum, but the solution will be better behaved, with less chance of absurdly large/small weights (like two weights of +1000 , -1000) that tend to occur with the un-regularized solution. As mentioned there, imposing a constraints that the weights have to be positive (no short selling) is also a mild form of regularization. So you have 3 easy ways to regularize, ridge, lasso and quadratic minimization with positive weights constraints).

Answer 2

Regularisation means that you impose structure on your problem; structure that could not be recognised from the data sample alone.

In the context of mean-variance optimisation, regularisation is mostly discussed when people estimate quantities from historical data (e.g. variances) and plug them into their objective function. (However, the term "regularisation" is not as commonly used in finance as in statistics, say.)

The trouble is that standard estimators can be driven by a few, extreme data points, which in turn mean that the model may consider a few specific ("lucky") assets as too good, which in turn leads to overly concentrated portfolios. If you only care about variance (i.e. ignore mean returns), the impact is less dramatic, but it is still there.

People have essentially dealt in two ways with this problem: either they constrained ("regularised") the inputs (e.g. the variance-covariance matrix), or they constrained the resulting portfolio, typically by imposing min/max-weights. With either approach, the result is that assets become "more equal", i.e. you enforce diversification and reduce large bets. (It can even be shown that both approaches are closely related; see Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps.)

Note that using a method such as the Lasso goes against this idea since it typically provides sparse solutions: it will rather reduce the number of assets, i.e. reduce diversification. (This may be useful, however, if you are interested in rebalancing a portfolio, in which case you may want only few trades.)

Instead, two approaches that have been used a lot in particular for mean-variance/minimum-variance are factor models and shrinkage.

In a factor model, you assume that all covariation is driven by common factors. So instead of directly estimating the full covariance matrix, you estimate factor models for all assets and then construct the covariance matrix from the estimated factor models.

Shrinkage essentially means that you replace the standard estimator for the variance--covariance matrix with a linear (convex) combination of the standard estimator and a more structured estimate. The latter may be a simple diagonal matrix, for instance; or it may be itself the variance--covariance matrix derived from a usually-simple factor model. A good start should be a paper such as www.ledoit.net/honey.pdf.

If you are completely free as to what regularisation means, I would start with alternative estimators (i.e. more-structured estimators) for the variance-covariance matrix.

In case you are also willing to use R: there exists several implementations of such estimators; a whole collection is in package RiskPortfolios. Even if you use Python, you may find the list of possibilities in the package of interest. A simple sketch (in R) for running a backtest on the mentioned dataset could look as follows:

library("RiskPortfolios")
library("NMOF") ## https://github.com/enricoschumann/NMOF
library("PMwR")
library("zoo")

P <- French("~/Downloads/French",  ## path where to store raw file
            dataset = "48_Industry_Portfolios_daily_CSV.zip", 
            weighting = "value", 
            frequency = "daily",
            price.series = TRUE,
            na.rm = TRUE)

## use data from January 2000
P <- window(zoo(P, as.Date(row.names(P))),
            start = as.Date("2000-01-01"),
            end   = as.Date("2019-01-31"))

signal_mv <- function(cov_fun, wmin, wmax, n, ...) {

    ## cov_fun .. takes a matrix R of returns 
    ##            (plus ...), and evaluates to 
    ##            the variance--covariance matrix
    ##            of those returns

    ## fetch prices for the last 10 years
    ## (~2500 trading days)
    P <- Close(n = 2500)

    ## compute returns: use only every nth price
    R <- returns(P[seq(1, nrow(P), by = n), ])
    minvar(cov_fun(R, ...), wmin, wmax)
}


## Backtest 1: Standard Covariance estimator
bt.mv <- btest(prices = list(coredata(P)),
               signal = signal_mv,
               do.signal = "lastofquarter",
               convert.weights = TRUE,
               initial.cash = 100,
               b = 2500,
               cov_fun = cov,
               wmin = 0.00,
               wmax = 0.2,
               n = 20,
               timestamp = index(P),
               instrument = colnames(P))
summary(as.NAVseries(bt.mv))
## [....]
## Return (%)             11.5  (annualised)
## Volatility (%)         10.1  (annualised)
## [....]

## Backtest 2: Covariance estimator with constant correlation
cov_const <- function(R, ...)
    covEstimation(R, list(type = "const"))

bt.mv <- btest(prices = list(coredata(P)),
               signal = signal_mv,
               do.signal = "lastofquarter",
               convert.weights = TRUE,
               initial.cash = 100,
               b = 2500,
               cov_fun = cov_const,
               wmin = 0.00,
               wmax = 0.2,
               n = 20,
               timestamp = index(P),
               instrument = colnames(P))
summary(as.NAVseries(bt.mv))
## [....]
## Return (%)             11.1  (annualised)
## Volatility (%)          9.8  (annualised)
## [....]

Answer 3

In your other post's Python code, you already have the sample covariance matrix as:

cov_matrix = returns.cov()

To also get the covariance used by ridge regression, put on the next line:

cov_ridge = cov_matrix + lamda*np.eye(N)

The second term increases the values along the diagonal of cov_matrix, using a $\lambda$ (lamda) regularization factor for ridge regression that you have to set, and identity matrix np.eye(N) for $N$ assets. If your returns dataset is $T\times N$ shaped, then N=np.shape(returns)[1], or if $N\times T$, then N=len(returns)). Ridge usually needs much higher values for $\lambda$ than lasso does to effect a noticeable difference.

Now, replace cov_matrix in any of your other main functions with the ridge covariance matrix cov_ridge in order for the rest of your code to derive ridge portfolio weights:

def portfolio_annualised_performance(weights, mean_returns, cov_ridge):

and

def random_portfolios(num_portfolios, mean_returns, cov_ridge, risk_free_rate):

Given that $(\Sigma+\lambda I)$ is cov_ridge, the ridge portfolio weights using the analytical solution for the GMV (global minimum variance) portfolio will now be:

$$\omega^{ridge} = \frac{(\Sigma+\lambda I)^{-1}\iota}{\iota^{\top}(\Sigma+\lambda I)^{-1}\iota}$$

As for lasso, there is no closed-form solution like ridge, so you will want to run a package like Sci-kit learn onto the Kempf and Memmel (2006) regression formula below using lasso's own $\lambda$ factor (input into the package's pre-built lasso function), then re-arrange the coefficients into lasso GMV portfolio weights.

$$\tilde{r}_n= \alpha +\beta_1 (\tilde{r}_n-\tilde{r}_1) + \beta_2 (\tilde{r}_n-\tilde{r}_2) +\cdots + \beta_{n-1}(\tilde{r}_n-\tilde{r}_{n-1})+\tilde{\epsilon}$$

$$\boldsymbol{\omega}^{lasso} = \begin{bmatrix} \hat{\beta}_1 \\ \vdots \\ \hat{\beta}_{N-1} \\ 1-\sum_{n=1}^{N-1}\hat{\beta}_{n} \end{bmatrix}$$ Sci-kit learn documentation for lasso can be found here. You can also use their ridge regression module using the same ridge $\lambda$ you used on the analytical shortcut I described earlier to get the same answer for ridge. Your code, however, is using the covariance approach to portfolio selection, which is fine for getting efficient frontiers, but if you want to use regularized regression for the GMV, create a new function solely for that purpose.