5
$\begingroup$

Expected Shortfall is not elicitable as some papers have pointed out. That simply means that there is no scoring function that elicits ES.

My question is, does this imply that Expected Shortfall point forecasts are impossible to backtest?

Still, there are backtests available for ES. Second question, how is the ES forecast backtested, that is how is the value of ES used in any backtest. To what is the ES forecast compared to in a backtest.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

I think it was T. Gneiting in 2011 who first proved that ES is not elicitable (Making and Evaluating Point Forecasts, Journal of the American Statistical Association Volume 106, 2011 - Issue 494) , which then threw some doubt as to whether it was backtestable. Carlo Acerbi pretty much put the matter to bed a few years ago in a number of papers, in which he explained that it does not matter for backtesting purposes whether or not ES is elicitable. Here is a link to his 2014 paper in which he gives three methods for backtesting ES.

Improve this answer
$\endgroup$
1
$\begingroup$

This is formally correct. However, I am not sure if practically it really makes any difference as Tasche points out: https://workspace.imperial.ac.uk/mathfin/Public/Seminars%202013-2014/Tasche_November2013_Slides.pdf

Edit: ES can be expressed as a weighted average of percentiles, which are backtestable as Bernoulli. Therefore backtest a handful of quantiles and you effectively backtest ES (Tasche says four, I believe FRTB says two). The point is that you don't backtest the ES number directly, but quantiles of the tail that generated your ES number.

Standard Deviation is not elicitable neither, but if you give me a sample I can tell you if the StdDev you gave me is good enough.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Well practically it does make a difference as the ES violations aren't observed Bernoulli distributed variables. I just don't understand how you're able to use a backtest for ES. That is, to what is the ES forecast compared to in a backtest like the one on page 9/10 of this paper papers.ssrn.com/sol3/papers.cfm?abstract_id=2514403? $\endgroup$
    – Eren
    Commented Mar 13, 2016 at 22:05
  • $\begingroup$ Please see edit $\endgroup$
    – Kiwiakos
    Commented Mar 14, 2016 at 0:39
  • $\begingroup$ If, for instance, some of the VaR levels are rejected while others are not, then this test gives you hard to interpret results. Therefore, I want to use the test on page 9 of that paper. Thank you for your answers, I'll see whether I am able to get that test up and running with the information and intuition you provided. In the end, does that mean that you don't even need to construct an ES forecast and still are able to backtest it? $\endgroup$
    – Eren
    Commented Mar 14, 2016 at 11:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.