# Which one is best Performance evaluation measures?

I want to compare the performance of various volatility models like GARCH, eGARCH, and gjrGARCH from actual volatility(computed using high frequency data). I found 3 common performance evaluation measures used profoundly in literature ie Root Mean Square Error(RMSE), Mean Absolute Error(MAE), and Mean Absolute Percentage Error(MAPE). But I found conflicting results from above 3 criteria, like I found RMSE is minimum for GARCH model, MAE is minimum for eGARCH model, and MAPE is minimum for gjrGARCH model.

I want to know which one is the best measures for evaluating performance and which results should I report?If all three are equally valid then how can I interpret the results? Is there any other way to compare the performance of various volatility models other than performance measures ?

• What is your proxy for the true volatility ? – Malick Nov 26 '15 at 12:02
• to be clear, do you evaluate the adequacy of your volatility estimation or do you evaluate the adequacy of forecasts ? – Malick Nov 26 '15 at 12:10
• @Malick I used high frequency data for true volatility and i want to evaluate the adequacy of forecast. – Neeraj Nov 26 '15 at 12:20

Before prediction you should see which models fit better your data. First before choosing a GARCH model or a GARCH type model with leverage efect you shoud perform the Engle-Ng sign bias test to see if the asset that your are modelling is affected by it, if yes a simple GARCH model won't be a good model. After the estimation and given that all parameters are significant compere the information criteria; for example the Akaike, Bayes or Hannan-Quinn and see which model has the smallest one. With this you should select the model and then perform a prediction, and you can evaluate the performance of it with a back test. I have personally done this and usually an APARCH with generalize hyperbolic distribution is the best if the data present leverage erect.

UPDATE

Sure this is what I did with r and the package rugarch for the stock of EXITO a Colombian company:

1) fit a GARCH(1,1):

garch11Spec=ugarchspec(variance.model = list(model='sGARCH',garchOrder=c(1,1)),mean.model=list(armaOrder=c(0,0)))
garch11Fit=ugarchfit(garch11Spec,rtn) ## rtn is the date which is daily last price


2) Engle-Ng sign bias for leverage effect:

signbias(garch11Fit)
t-value               prob      sig
Sign Bias          0.19343098 0.8466364
Negative Sign Bias 0.15413313 0.8775166
Positive Sign Bias 0.02615509 0.9791356
Joint Effect       0.06291070 0.9958816


see that all p-values are higher than 0.1 son you don't reject the null hypothesis so there is no leverage effect

3) Look at the coefficients (the robust version) and there p-value, you will expect that all of theme to be statistically significant

garch11Fit@fit$robust.matcoef Estimate Std. Error t value Pr(>|t|) mu 3.735730e-05 1.237313e-04 0.3019229 7.627109e-01 omega 1.238605e-05 5.156212e-08 240.2160962 0.000000e+00 alpha1 2.517465e-01 5.271207e-02 4.7758795 1.789235e-06 beta1 5.822453e-01 3.733553e-02 15.5949369 0.000000e+00  4) Information criteria: infocriteria(garch11Fit) Akaike -6.941621 Bayes -6.932700 Shibata -6.941626 Hannan-Quinn -6.938391  you want the smallest value possible Here are the 4 steps together for the aparch: signbias(aparchFit) t-value prob sig Sign Bias 0.17982812 0.8573014 Negative Sign Bias 0.05186043 0.9586438 Positive Sign Bias 0.57709943 0.5639217 Joint Effect 0.35736070 0.9489028 aparchFit@fit$robust.matcoef
Estimate   Std. Error   t value     Pr(>|t|)
mu     0.0000186132 0.0001385818 0.1343120 8.931558e-01
omega  0.0019649457 0.0004959927 3.9616428 7.443584e-05
alpha1 0.2521930795 0.0433561290 5.8167804 5.999189e-09
beta1  0.5734338681 0.0809929461 7.0800470 1.441069e-12
**gamma1 0.0482664316 0.0886992553 0.5441583 5.863326e-01**
delta  1.0000000000           NA        NA           NA


you can see, gamma1 which is the leverage effect parameter is not significant, p-value = 0.58 as expected, so not a good model

infocriteria(aparchFit)

Akaike       -6.942806
Bayes        -6.931655
Shibata      -6.942813
Hannan-Quinn -6.938769


if you compera the criteria with the GARCH(1,1) you can se that the Bayes and the Hannan-Quinn are smaller in the first model, the other 2 smaller for the aparch but YOU KNOW that the aparch isn't the right model because there is NO leverage effect, can't just trust the information criteria you have to understand the models to select one. With this stock is easy because it has no leverage effect so you have to stick with a GARCH(1,1) if you get leverage effect should perform this with the different model that you want to use to find the best.

If after doing this with the differnt models you don't have a clear winner do a back test on the prediction and compere the RMSE, MSE or MAE and also do a VaR backtest as the other comment says

• thanks for showing your interest in the question. Can you please elaborate how you made comparison between various models and how you choosed best one ? – Neeraj Dec 7 '15 at 5:53

If you are interested in evaluating forecasts accuracy, you could compare Value-at-risk forecasts. It has the advantage to take into account the forecast density (via quantile). Then you can compare easily their forecast accuracy via the Kupiec test for instance. Because if you just use points forecasts as it seems you are doing your results won't take the uncertainty of your point forecasts into account. In this case the ranking of your models will highly depends of the loss function specified and of actual realizations.

Additionally you can also have a look to the Mincer-Zarnowitz regression, which is easy to implement (Mincer, J., and V. Zarnowitz (1969): The Evaluation of Economic Forecasts Economic Forecasts and Expectations. in J.Mincer, New York: National Bureau of Economic Research.)

So I think you'll get more stable results if you stick to VaR forecasts (ie point forecasts + the confidence interval) instead of comparing point forecasts.

• thanks @Malick. Can you suggest some papers that have used techniques like VaR and Kupeic test to evaluated the performance of the forecasting accuracy ? – Neeraj Nov 27 '15 at 6:49
• @Neeraj sure : Giot, P., & Laurent, S. (2004). Modelling daily Value-at-Risk using realized volatility and ARCH type models. Journal of Empirical Finance, 11(3), 379–398. doi.org/10.1016/j.jempfin.2003.04.003 – Malick Nov 27 '15 at 10:31
• Thanks @Malick. I will respond quickly after reading this paper. – Neeraj Nov 28 '15 at 7:09