# Understanding out-of-sample performance metrics for Realized Volatility

I fitted several models on a realized volatility process and then proceeded to obtain out-of-sample results. I'm struggling to interpret these results apart from to tell model A seems better than model B. In particular, I am looking to answer the question: "Does model A have satisfactory forecasting performance?".

I seem to gravitate towards MAPE and MZ-R2 (Mincer-Zarnowitz R-squared). These seem to be absolute measures. I am not experienced enough to trust my judgement however.

Updated: Diebold-Mariano table.

              ARFIMA(0,d,0) HAR(1,5,22) HAR(1,4,30) S-GARCH(1,1) GJR-GARCH E-GARCH REAL-GARCH
ARFIMA(0,d,0)       1.00000     0.04714     0.04828      0.00005   0.00004 0.00000    0.48256
HAR(1,5,22)         0.04714     1.00000     0.29521      0.05598   0.05561 0.05718    0.01175
HAR(1,4,30)         0.04828     0.29521     1.00000      0.03824   0.03875 0.04002    0.01156
S-GARCH(1,1)        0.00005     0.05598     0.03824      1.00000   0.62604 0.81480    0.00031
GJR-GARCH           0.00004     0.05561     0.03875      0.62604   1.00000 0.73391    0.00030
E-GARCH             0.00000     0.05718     0.04002      0.81480   0.73391 1.00000    0.00002
REAL-GARCH          0.48256     0.01175     0.01156      0.00031   0.00030 0.00002    1.00000

• You cannot make a satisfactory conclusion from the out-of-sample loss functions between the models, since the marginal difference might be attributable to the sample size, length and time-interval. This is the reason you use forecast comparison methods: testing the statistical significance of the out-of-sample performance between the models. Above, the Mincer Zarnowitz regression would be your best choice of comparison method though.
– Pleb
Aug 23 at 15:47
• @Pleb The models all estimate the volatility process between the same dates and all use the same data. For example, REAL-GARCH gets the same RV series as ARFIMA() and HAR-RV. GARCH models estimate the volatility for the same dates in the RV process which is used as the "realized" volatility against which GARCH output can be compared.
– s5s
Aug 23 at 16:08
• Surely, but you haven't tested whether the out-of-sample loss differences are statistically different from 0 and thus I wouldn't not make the conclusion that one model is better than the other.
– Pleb
Aug 23 at 16:17
– s5s
Aug 23 at 16:25
• Yes, I'll do an answer and give you some literature you can read :-)
– Pleb
Aug 23 at 16:25

You can compare the losses against each model and determine the "best" model to be the one with the smallest losses. In many cases for larger studies, the results might be ambiguous where one or more models are favored by different loss functions. Therefore, we want to know if we can construct statistical tests that evaluates the significance of the performance, based on the loss differences between the models. It turns out that we can, and it has been studied extensively within academia.

# Forecast comparison analysis:

When we want to do a forecast comparison analysis, there is one thing we want to do: testing whether the forecast difference is statistically different from zero. If we let $$L(\theta_t, \Sigma_{it})$$ be your loss function (just think QLIKE or MSE) with $$\Sigma_{it}$$ be your covariance estimate for model $$i$$ at time $$t$$ and $$\theta_t$$ is your robust proxy (since volatility is latent). Then the loss difference can be defined as $$d_{ij,t}=L(\theta_t, \Sigma_{it}) - L(\theta_t, \Sigma_{jt})$$ with a corresponding constructed null hypothesis:

$$H_0: \quad \mathbb{E}\left[d_{ij,t}\right] = 0, \qquad \forall \: t.$$

Constructing the test-statistic, we need the standard error which is usually found by bootstrapping the loss-differences and then calculating the bootstrapped standard error (Intuitively and under enough regularity, the bootstrap standard error will be close to the population standard error). Be aware that the (composite) null hypothesis can differ slightly between each comparison method.

This is the starting point of many forecast comparison tests including the well-known Diebold-Mariano test, which is a pair-wise forecast comparison test. There exists a plephora of different forecast comparison methods, where most of them focus on comparing multiple out-of-sample forecasts at once. Before listing most of them, I want you to consider reading two articles that investigates different volatility models (measures) using forecast comparison methods:

The theoretical background for many of the forecast comparison methods is quite extensive and hard to comprehend. Thus, people tend to pick a few of the methods and get far by understanding the intuition and results behind them. From the many articles I have read, the Diebold-Mariano test is the favorite for pair-wise comparison and The Model Confidence Set for multiple forecast comparison.

# Listing different forecast comparison methods:

• Diebold-Mariano test: Diebold, Francis X., and Robert S. Mariano (2002). "Comparing predictive accuracy." They made an updated paper responding to their original paper by detailing the use and abuse of the test.

• White's test (Reality check or RC): White, Halbert (2000). "A reality check for data snooping." As described in the first article above, the RC test "lacks" in power and has a hard time distinguishing between "good" and "bad" forecasts.

• Hansens Superior Predictive Ability (SPA): Hansen, Peter Reinhard (2005). "A test for superior predictive ability." This forecast comparison method is more robust to the inclusion of poor forecast in contrast to RC.

• Romano-Wolf test: Romano, Joseph P., and Michael Wolf (2005). "Stepwise multiple testing as formalized data snooping." This is similar to SPA, however the primary difference is that the Romano-Wolf test identifies the set of models that are better than the benchmark, whereas the SPA asks the question whether any forecast is better than the benchmark.

• The Model Confidence Set (MCS) Hansen, Peter R., Asger Lunde, and James M. Nason (2011). "The model confidence set." Intuitively, this comparison test gives you a set of models within a given level of "confidence" where the selected models are "equal" in out-of-sample predictive ability. In my opinion, this is the most complicated forecast comparison method to understand from a theoretical perspective. In that regard, the authors also made another paper posing as a "guide" on using the MCS method and how to choose the best models. Here, they also lay an intuitive foundation for the test.

• Multi-Horizon SPA & MCS: R. Quaedvlieg (2020), "Multi-Horizon Forecast Comparison." In this recent article, R. Quaedvlieg provide extensions of the SPA & MCS tests to jointly compare multiple horizon forecasts. He further concludes that the tests lead to more coherent results. Comparing model forecasts at many individual horizons independently, will implicitly give us a multiple testing problem that leads to more type 1 errors ie. false rejections of the null (implying that models can significantly differ at a given forecast horizon). As such, in finite samples we are likely to find that a mis-specified model will outperform even the population model at one of the many horizons one could consider. Comparing all horizons jointly guards us against this problem. The multi-horizon tests are not restricted to economics, but can also be used to compare climate forecasts at different horizons et cetera.

Kevin Sheppard has provided good graphical illustrations of the SPA, Romano-Wolf, and MCS tests in a Python notebook. They are also implemented in his arch package.

I am by no means an expert on the forecast comparison methods detailed above. However, you can get far by understanding the intuition behind the methods (pick one or two out) and how to interpret the output. Understanding the theoretical results behind the tests are not needed in order to use them. I hope this helps.

• I should probably ask this as a separate question. If DM(modelA, modelB) > 0.05 and DM(modelB, modelC) > 0.05 but DM(modelA, modelC) < 0.05 then would one conclude that modelA and modelC have the same predictive power?
– s5s
Aug 23 at 19:09
• @s5s You're making isolated pairwise tests. Thus IMO, the interpretation will only lie on the DM(model A, model C). AFAIK, assuming you have many models, and make many DM-tests (multiple tests), then you'll not control for type 1 errors which will lead to more false rejections of the null. Many multiple comparison methods control for type 1 errors by controlling the familywise error rate FWER, including the MCS (briefly specified in the updated DM paper). Currently, this is the best answer I can give you. For a bulletproof answer you should ask on Stats.SE
– Pleb
Aug 23 at 21:30