Which one is best Performance evaluation measures?

Question

I want to compare the performance of various volatility models like GARCH, eGARCH, and gjrGARCH from actual volatility(computed using high frequency data). I found 3 common performance evaluation measures used profoundly in literature ie Root Mean Square Error(RMSE), Mean Absolute Error(MAE), and Mean Absolute Percentage Error(MAPE). But I found conflicting results from above 3 criteria, like I found RMSE is minimum for GARCH model, MAE is minimum for eGARCH model, and MAPE is minimum for gjrGARCH model.

I want to know which one is the best measures for evaluating performance and which results should I report?If all three are equally valid then how can I interpret the results? Is there any other way to compare the performance of various volatility models other than performance measures ?

Answer 1

Before prediction you should see which models fit better your data. First before choosing a GARCH model or a GARCH type model with leverage efect you shoud perform the Engle-Ng sign bias test to see if the asset that your are modelling is affected by it, if yes a simple GARCH model won't be a good model. After the estimation and given that all parameters are significant compere the information criteria; for example the Akaike, Bayes or Hannan-Quinn and see which model has the smallest one. With this you should select the model and then perform a prediction, and you can evaluate the performance of it with a back test. I have personally done this and usually an APARCH with generalize hyperbolic distribution is the best if the data present leverage erect.

UPDATE

Sure this is what I did with r and the package rugarch for the stock of EXITO a Colombian company:

1) fit a GARCH(1,1):

garch11Spec=ugarchspec(variance.model = list(model='sGARCH',garchOrder=c(1,1)),mean.model=list(armaOrder=c(0,0)))
garch11Fit=ugarchfit(garch11Spec,rtn) ## rtn is the date which is daily last price

2) Engle-Ng sign bias for leverage effect:

signbias(garch11Fit) 
 t-value               prob      sig
Sign Bias          0.19343098 0.8466364    
Negative Sign Bias 0.15413313 0.8775166    
Positive Sign Bias 0.02615509 0.9791356    
Joint Effect       0.06291070 0.9958816 

see that all p-values are higher than 0.1 son you don't reject the null hypothesis so there is no leverage effect

3) Look at the coefficients (the robust version) and there p-value, you will expect that all of theme to be statistically significant

garch11Fit@fit$robust.matcoef
       Estimate   Std. Error     t value     Pr(>|t|)
mu     3.735730e-05 1.237313e-04   0.3019229 7.627109e-01
omega  1.238605e-05 5.156212e-08 240.2160962 0.000000e+00
alpha1 2.517465e-01 5.271207e-02   4.7758795 1.789235e-06
beta1  5.822453e-01 3.733553e-02  15.5949369 0.000000e+00

4) Information criteria:

infocriteria(garch11Fit)
Akaike       -6.941621
Bayes        -6.932700
Shibata      -6.941626
Hannan-Quinn -6.938391

you want the smallest value possible

Here are the 4 steps together for the aparch:

signbias(aparchFit)
                      t-value      prob sig
Sign Bias          0.17982812 0.8573014    
Negative Sign Bias 0.05186043 0.9586438    
Positive Sign Bias 0.57709943 0.5639217    
Joint Effect       0.35736070 0.9489028 

aparchFit@fit$robust.matcoef
           Estimate   Std. Error   t value     Pr(>|t|)
mu     0.0000186132 0.0001385818 0.1343120 8.931558e-01
omega  0.0019649457 0.0004959927 3.9616428 7.443584e-05
alpha1 0.2521930795 0.0433561290 5.8167804 5.999189e-09
beta1  0.5734338681 0.0809929461 7.0800470 1.441069e-12
**gamma1 0.0482664316 0.0886992553 0.5441583 5.863326e-01**
delta  1.0000000000           NA        NA           NA

you can see, gamma1 which is the leverage effect parameter is not significant, p-value = 0.58 as expected, so not a good model

infocriteria(aparchFit)

Akaike       -6.942806
Bayes        -6.931655
Shibata      -6.942813
Hannan-Quinn -6.938769

if you compera the criteria with the GARCH(1,1) you can se that the Bayes and the Hannan-Quinn are smaller in the first model, the other 2 smaller for the aparch but YOU KNOW that the aparch isn't the right model because there is NO leverage effect, can't just trust the information criteria you have to understand the models to select one. With this stock is easy because it has no leverage effect so you have to stick with a GARCH(1,1) if you get leverage effect should perform this with the different model that you want to use to find the best.

If after doing this with the differnt models you don't have a clear winner do a back test on the prediction and compere the RMSE, MSE or MAE and also do a VaR backtest as the other comment says

Answer 2

If you are interested in evaluating forecasts accuracy, you could compare Value-at-risk forecasts. It has the advantage to take into account the forecast density (via quantile). Then you can compare easily their forecast accuracy via the Kupiec test for instance. Because if you just use points forecasts as it seems you are doing your results won't take the uncertainty of your point forecasts into account. In this case the ranking of your models will highly depends of the loss function specified and of actual realizations.

Additionally you can also have a look to the Mincer-Zarnowitz regression, which is easy to implement (Mincer, J., and V. Zarnowitz (1969): The Evaluation of Economic Forecasts Economic Forecasts and Expectations. in J.Mincer, New York: National Bureau of Economic Research.)

So I think you'll get more stable results if you stick to VaR forecasts (ie point forecasts + the confidence interval) instead of comparing point forecasts.