How do I evaluate the suitability of a GARCH model?

Question

Suppose I downloaded the closing price of a company, say Google or whatever, I want to use GARCH model to model and forecast the volatility of the return.

To simplify, I only have two questions.

1. As we know, GARCH is used to predict volatility. But, after we get the new return(or price) data, how well is the GARCH prediction? Is there any quantitative method to evaluate this?

2. Once we decided to use GARCH(p,q), how do we choose the order p and q?(for example, if we choose (p,q)=(1,1), then why we do not choose(2,2) or whatever)

To be more specific,

1. If I use GARCH(1,1) to model the returns, how do we know that the result fit the real data very well? Is there any way to evaluate this thing? (Is this the goodness-of-fit problem? )

2. When I was thinking about the previous problem, a new one came to me. Suppose I use the ARMA-GARCH model to model the return data. ARMA is to model the return, and GARCH to model the volatility. In this way, how do we evaluate the fitted ARMA-GARCH result? Suppose I use the data up to last week to forecast the return and its volatility in this week. Then, I can use this week's price to calculate real return, and then compare with the predicted return by ARMA, to see how well it works. BUT, how do we know how well is the predicted volatility by GARCH???I mean, the return in this week is a random variable in the view of last week, and it has a standard deviation, which is the volatility, and we can use the GARCH to forecast it. However, after I know the real return in this week, it becomes a constant, and there is no volatility since there is no randomness. Thus, I do not have the real volatility...In this way, how do I evaluate the result of GARCH? I do not have a standard to compare with.

3. How do we determine the orders of GARCH. Once I download the data, how do I know whether I need to use ARMA(1,1)-GARCH(1,1) or ARMA(20,30)-GARCH(40,50)? Or, is there any theory or function in Matlab help us to do the selection?

Answer 1

Which model to choose from a pool of candidate models depends on what you want to do with it.

If you want to do forecasting, you should select a model that would be expected to deliver the most accurate forecasts. Akaike's information criterion (AIC) is known to asymptotically select a model (from a given pool) that will deliver the forecasts with the lowest mean squared error. Therefore, it is often used as a model selector in forecasting exercises, and justifiably so. AIC is easy to obtain from a fitted model (by hand or via a built-in function) , so it makes for a handy tool. An alternative to AIC could be the focused information criterion (FIC), which allows for much more flexibility in specifying what kind of loss function you wish to minimize (e.g. mean absolute error in place of mean squared error), but it also requires more involvement from the researcher; you need to derive the specific version of FIC based on your loss function before you can apply it.

If you want to recover the true model and the true model happens to be among the set of candidate models, use Bayesian information criterion (BIC). Asymptotically it will select the true model with probability=1. Since most of the time the true model is not in the pool (the reality is more complex than our models), BIC will tend to select a model that approximates the true model the most closely.

If you have yet another goal, a philosophically straightforward way to do model selection is to derive a version of FIC that reflects your goal (but again, it may be a challenging exercise on its own).

In all these cases, the candidate model pool has to be selected in the first place, and that is not a trivial task either. You may bring in subject-matter knowledge; probably you have an idea what type of models should describe your data generating process well. You may then explore a pool as large as is computationally feasible. However, bear in mind that information criteria are not entirely robust against overfitting. When you have a number of models with the same number of parameters, both AIC and BIC reduce to maximizing the likelihood, and that is not at all robust against overfitting. So in practice you may try a somehwta large pool but without going to extreme sizes.

---

The practice of seeking a model with statistically significant coefficients and well-behaved residuals as suggested in another answer is somehow tangential to the approach above. I am not sure the latter strategy optimizes any sensible loss function except for delivering a nice in-sample fit (and I mean: I am not sure). However, in sample we already know everything, we know each data point. The real goal is normally to generalize out of sample and either say something about the properties of the data generating process (e.g. test some hypotheses about it) or do forecasting. So then the approach above based on AIC, BIC and FIC makes sense. (Even so, recall that once model selection has been performed and you have ended up with a model of choice, you cannot trust the significance tests anymore because they are conditional on the model having been selected. The way of accounting for this conditioning properly is becoming an active research area in statistics.)

---

And then I have not touched upon regularized estimation and model averaging which are more advanced approaches that generalize the problem of model selection.

Also, you may find much more about all this at Cross Validated, a sister site in the Stack Exchange system.

Answer 2

The general procedure is to start out simple, real simple, and build your model up only as necessary. AR(q), q=0 to start with. Test the lagged autocorrelations of the error terms, and increase q until they are no longer significant. Test for ARCH, and if it's significant, you have an ARCH(q) model. Then move on with GARCH(1,q), GARCH(2,q), and when the GARCH errors are no longer significant you have GARCH(p,q) where increasing p or q would have little additional explanatory power. Perhaps you could even reduce q with GARCH as compared to ARCH. Test and see if it's significant.

Every one of these models has standard tests for errors. Test and make sure it's significant each time before you throw in a new effect or increase the order of the model. This is somewhat subjective; it's not a canned procedure.