Regularisation means that you impose structure on your
problem; structure that could not be recognised from the
data sample alone.
In the context of mean-variance optimisation,
regularisation is mostly discussed when people estimate
quantities from historical data (e.g. variances) and
plug them into their objective function. (However, the term "regularisation" is not as commonly used in finance as in statistics, say.)
The trouble is
that standard estimators can be driven by a few,
extreme data points, which in turn mean that the model
may consider a few specific ("lucky") assets as too
good, which in turn leads to overly concentrated
portfolios. If you only care about variance (i.e. ignore mean returns), the impact is less dramatic, but it is still there.
People have essentially dealt in two ways with this
problem: either they constrained ("regularised") the
inputs (e.g. the variance-covariance matrix), or they
constrained the resulting portfolio, typically by
imposing min/max-weights. With either approach, the
result is that assets become "more equal", i.e. you
enforce diversification and reduce large bets. (It can even be shown
that both approaches are closely related; see Risk Reduction in Large Portfolios: Why Imposing the Wrong Constraints Helps.)
Note that using a method such as the Lasso goes against
this idea since it typically provides sparse solutions: it will rather reduce the number of assets,
i.e. reduce diversification. (This may be useful,
however, if you are interested in rebalancing a
portfolio, in which case you may want only few trades.)
Instead, two approaches that have been used a lot in
particular for mean-variance/minimum-variance are
factor models and shrinkage.
In a factor model, you assume that all covariation is
driven by common factors. So instead of directly estimating the full covariance matrix, you estimate factor models for all assets and then construct the covariance matrix from the estimated factor models.
Shrinkage essentially means
that you replace the standard estimator for the
variance--covariance matrix with a linear (convex)
combination of the standard estimator and a more
structured estimate. The latter may be a simple
diagonal matrix, for instance; or it may be itself the
variance--covariance matrix derived from a
usually-simple factor model. A good start should be a paper such as www.ledoit.net/honey.pdf.
If you are completely free as to what regularisation means, I would start with alternative estimators (i.e. more-structured estimators) for the variance-covariance matrix.
In case you are also willing to use R: there exists several implementations of such estimators; a whole collection is in package RiskPortfolios
. Even if you use Python, you may find the list of possibilities in the package of interest. A simple
sketch (in R) for running a backtest on the mentioned dataset could look as follows:
library("RiskPortfolios")
library("NMOF") ## https://github.com/enricoschumann/NMOF
library("PMwR")
library("zoo")
P <- French("~/Downloads/French", ## path where to store raw file
dataset = "48_Industry_Portfolios_daily_CSV.zip",
weighting = "value",
frequency = "daily",
price.series = TRUE,
na.rm = TRUE)
## use data from January 2000
P <- window(zoo(P, as.Date(row.names(P))),
start = as.Date("2000-01-01"),
end = as.Date("2019-01-31"))
signal_mv <- function(cov_fun, wmin, wmax, n, ...) {
## cov_fun .. takes a matrix R of returns
## (plus ...), and evaluates to
## the variance--covariance matrix
## of those returns
## fetch prices for the last 10 years
## (~2500 trading days)
P <- Close(n = 2500)
## compute returns: use only every nth price
R <- returns(P[seq(1, nrow(P), by = n), ])
minvar(cov_fun(R, ...), wmin, wmax)
}
## Backtest 1: Standard Covariance estimator
bt.mv <- btest(prices = list(coredata(P)),
signal = signal_mv,
do.signal = "lastofquarter",
convert.weights = TRUE,
initial.cash = 100,
b = 2500,
cov_fun = cov,
wmin = 0.00,
wmax = 0.2,
n = 20,
timestamp = index(P),
instrument = colnames(P))
summary(as.NAVseries(bt.mv))
## [....]
## Return (%) 11.5 (annualised)
## Volatility (%) 10.1 (annualised)
## [....]
## Backtest 2: Covariance estimator with constant correlation
cov_const <- function(R, ...)
covEstimation(R, list(type = "const"))
bt.mv <- btest(prices = list(coredata(P)),
signal = signal_mv,
do.signal = "lastofquarter",
convert.weights = TRUE,
initial.cash = 100,
b = 2500,
cov_fun = cov_const,
wmin = 0.00,
wmax = 0.2,
n = 20,
timestamp = index(P),
instrument = colnames(P))
summary(as.NAVseries(bt.mv))
## [....]
## Return (%) 11.1 (annualised)
## Volatility (%) 9.8 (annualised)
## [....]