What type of analysis is appropriate for assessing the performance time-series forecasts?

Question

When using time-series analysis to forecast some type of value, what types of error analysis are worth considering when trying to determine which models are appropriate.

One of the big issues that may arise is that successive residuals between the 'forecast' and the 'realized' value of the variable may not be properly independent of one another as large amounts of data will be reused from one data point to its successive one.

To give an example, if you fit a GARCH model to forecast volatility for a given time horizon, the fit will use a certain amount of data, and the forecast is produced and then compared to whatever the realized volatility was observed for the given period of time, and it is then possible to find some kind of 'loss' value for that forecast.

Once everything moves forward a time period, assuming we refit (but even if we reuse the data parameters), there will be a very large overlap in all the data for this second forecast and realized volatility.

Since it is common to desire a model that minimises these 'losses' in some sense, how do you deal with the residuals produced in this way? Is there a way to remove the dependency? Are successive residuals dependent, and how could this dependency be measured? What tools exist to analyse these residuals?

Answer 1

I think you're looking for some way to test for autocorrelation in your residuals. If your model is good -- let's say you have an ARMA(1, 1) model for your forecast -- then the residuals from this model will be white noise. Which is to say that the difference between your forecast and the realization can not be predicted any better. The residual is some zero mean normally-distributed error.

Let's pick an extreme example. If your residual (the diff between forecast and realized) were always 1, then the residuals would be autocorrelated. Clearly if your model is always off by 1, then you can do better. So if the residuals in your model are autocorrelated, then you can do better.

The standard test for this these days in Ljung-Box, but in the past Box-Pierce and Durbin-Watson were also used.

Answer 2

I am not sure I clearly understand your question. But definitely you can do some analysis on the residuals, especially autocorrelation. If you find any significant autocorrelation, I suggest you add a ARMA process to your model to increase the accuracy of your forecast.

Answer 3

I am not sure I understand your question, but you might be conflating two different things:

1. autocorrelation in model residuals in a fixed sample (window) and
2. autocorrelation in forecast errors across samples (rolling or expanding windows).

(1) is undesirable as it indicates the model misses a pattern which it should ideally capture. This can be remedied, for example, by changing the model. One may add an ARMA structure to the model's error term (to get ARMA-GARCH from pure GARCH, for example), change the model's autoregressive order, or do some other changes.

(2) can happen by construction and need not indicate any problem with the forecasts or the modelling scheme that is generating them. Indeed, forecast errors of $h$ steps ahead will necessarily be MA($h-1$) processes; see e.g. Diebold "Forecasting in Economics, Business, Finance and Beyond" Chapter 10 "Point Forecast Evaluation", section 10.1 "Absolute Standards for Point Forecasts" (version of 14 December 2015; the linked version might change over time).