I am trying to predict the realised daily close to close variance of an equity index.
I checked the literature on volatility forecasting and tried a bunch of things on a dataset for the S&P 500. The most promising approached were an EWMA, GARCH and just using the squared VIX. I measured the performance of my predictions by looking at average absolute error, mean squared error and also doing a regression between my predictor and the actual squared returns. I am aware that squared returns are very noisy, so I also took longer term averages of them and checked how well my predictors forecasts these averages.
EWMA and GARCH showed a similar performance. I was surprised to see that the VIX did much better. I can clearly see that VIX is biased, since there is the volatility risk premium. Implied vol is on average higher than realised vol, since option sellers want to be compensated, but of course there are exceptions. I tried to remove the volatility risk premium from the VIX by subtracting some rolling averages of the realised risk premium, hoping that this would remove the bias, but my estimation got much worse afterwards.
Does anyone have some experience on this type of problem? Can you confirm my observations? Are there other methods I could try?