VaR backtesting. Reasons for over- and underestimation of value at risk estimates?

Question

I use an ARMA-GARCH model for univariate distributions and a copula model for the multivariate distribution. For the value at risk (VaR) estimation, I do a Monte Carlo simulation. I'm looking for some general reasons why VaR estimations over- or underestimate the VaR in the sense of this answer. And what would be some source to look that up?

Answer 1

I think there could be a few theoretical reasons for it.

1. VaR is distribution dependent. Even if bootstrapping, it is necessary that yhe underlying distribution satisfy second order conditions for convergence.
2. Also depends on the type of VaR being used. CVaR and var for instance capture two different things. CVaR is known to outperform VaR.

Answer 2

Here are some suggestions/random thoughts based on my own past experience debugging unexpected VaR values. They may help with Expected Shortfall (ES) as well.

Maybe the markets really behaved differently in-sample - the historical days that you use directly to calculate VaR or to calculate a covariance matrix - than out-of-sample - the days when you backtest the VaR v the P&L. Many VaR users exclude from their in-sample those days when some market factor moved more than some relatively large number of its historical standard deviations. To compensate, such atypical extreme shocks should be included among historical market stress scenarios instead, rather than in the VaR historical data.

Check whether the calculation uses punitive high volatilities for unrecognized market factors. In production environments, many VaR calculations sometimes encounter exposures to market factors that are not immediately recognized, for example the first time someone traded Bitcoin futures or a Kazakhstan cross-currency swap. :) Some "academic IT" calculators just stop right there, throw exceptions, and wait for the problems to be resolved the next day. But practically, some users of the VaR are eager to know how the new trades affected their VaR, even if the estimates err on the conservative side. Better engineered VaR calculators assume conservatively that the unrecognized factors have some very large volatility using some "wildcard" rules and no correlation to anything else, warn loudly that a data issue needs to be resolved. The "punitive" volatility needs to be large enough to incentivize VaR users to quickly replace it by historical volatilities, presumably lower, and correlations in the covariance matrix. If the warnings are not heeded, the punitive volatilities continue to be used, and the VaR is conservatively overstated.

Ideally, some new product approval process should try to ensure that the new market factors are known to the VaR calculator before they are first traded, but this isn't always practical.

Relatedly, market factors may be mislabeled, for example a senaitivity to a B-rated credit spread may be mislabeled as riskier unrated credit spread.

Monte Carlo perturbing market factors using pairwise correlations rather than principal components.

If you view the rates at various tenors of an intetrest rate curve as the market factors, and naively calculate their volatilities and correlations, and use Monte Carlo to generate some market movements, and look closely at the generated market movements, then many may strike you as not being realistic, and for some exposures, may give rise to unrealistic simulated P&L. The other extreme would be for the Monte Carlo to assume that rates are perfectlt correlater and change only in parallel. Perturbing (a sufficient number of) historical principal components is a better methodology.

Covariance matrix too sparse. If a portfolio is long two assets assumed to have correlation close to -1, or long one and short another assumed to have correlation close to 1, but the civariance matrix has 0 correlation, then the VaR would be overstated. A debugging tool useful to detect the absence of correlation is to have the VaR calculator print out its market factors in groups, with zero correlations between groups and non-zero correlations within groups.

Component VaR is another very helpful tool. Disaggregate the portfolio into as small pieces as possible to compare their VaR and Component VaR with their P&L, to see which pieces have VaR not matching P&L, and what they have in common. Disaggregate the VaR into component VaR by market factor types and individual market factors ans compare with Risk-Theoretical P&L (RTPL) attributing the P&L to the market factor. Large unexplained P&L (UPL) not explained by RTPL warrant investigation.

Whether the VaR calculator uses historical or Monte Carlo, it should be able to output the market scenarios that gave rise to P&L=VaR and the entire tail of P&Ls worse than VaR, filtering only the market factors that affect the portfolio under investigation, and showing their movements both in absolute terms and as the number of historical standard deviations. If these scenarios don't look realistic, then, perhaps, some historical dates need to be excluded from in-scope, or the covariance matrix needs to be more reality-like.