# Negative high frequency intraday volatility - Zhou estimator

To estimate high frequency tick data stock intraday volatility, I have read Robert Almgren's notes7.pdf

http://www.cims.nyu.edu/~almgren/timeseries/notes7.pdf

where he talks about the bias free estimator by Zhou:

$Z = \sum ((y_j - y_{j-1})^2 + 2(y_j - y_{j-1})(y_{j+1} - y_j))$

where $y_j$ is the log return of the price at time $j$.

However, this expression sometimes yields negative volatility. We see that the first term is a square which is always positive, but the second term $2(y_j-y_{j-1})(y_{j+1}-y_j)$ can be negative. So how do I treat this estimator? I want a positive volatility, not a negative! I am thinking of just applying absolute value on the second term, but that does not sound right. Any suggestions?

• Do you have a functional link to that Robert Almgren note? Thanks.
– Hans
Sep 15 '20 at 18:15

In any finite sample, it is always possible for the Zhou estimator to return a negative number, even though we know the unobservable parameter being estimated is non-negative. This is a well known issue in the academic literature. There are several approaches to dealing with this problem:

1) Ignore it. (I don't like this one). It is particularly nefarious if you ever move to a multivariate setting and you suddenly end up with a covariance matrix that is not positive semi-definite. (try taking the inverse of it and watch everything go to hell in your code).

2) Set your final estimator equal to $max(v, 0)$, where $v$ is the Zhou estimator. In practice, there is nothing wrong with this ad hoc estimator, although a pure theorist might get cross about the fact that the asymptotic distribution is affected. From a coding perspective, you might even want to use $max(v, \alpha)$ for some small positive constant $\alpha$ so you don't accidentally end up dividing by zero anywhere. NOTE: please do not take the absolute value of the autocovariance portion of the estimator, as you mention in the question. This makes no sense from an estimation perspective and will result in a heavily biased estimator that is inconsistent even under ideal modelling assumptions.

3) Use a different intraday-data based estimator of volatility that doesn't suffer from this problem. I strongly recommend this option. The Zhou estimator was state-of-the-art in 1996 (and that paper itself was astonishingly pre-scient - it took another decade for everyone else to catch on to the problems Zhou tried to solve in that paper), but a lot of work since then has demonstrated that it will be heavily biased for many high-frequency datasets, see e.g. Hansen, Lunde (2006) "Realised Variance and Market Microstructure Noise". Probably the most popular estimator at the present point in time is the realised kernels estimator of Barndorff-Nielsen, Hansen, Lunde, and Shephard. The original 2008 paper is a bit heavy going for a non-theorist, but have a look at Barndorff-Nielsen, Hansen, Lunde, and Shephard (2009) "Realized Kernels in Practice" - it is much more friendly. (I have an implementation of this estimator in the Julia language based off this paper).

This estimator also has the nice property that if you use a Parzen kernel (see the above reference for more detail), then the estimator will always be non-negative. Be warned, for other types of kernel function, especially flat-top kernels, there is no guarantee of non-negativity. See eg footnote 2 of the above-mentioned paper.

If you want something simpler in that you can implement quickly, then just use a low sampling frequency realized variance, e.g. 5-minute realized variance or maybe 10-minute. Since this is just a sum of squared intrady returns, it is guaranteed to always be non-negative and will, in my experience, provide a better estimate than the Zhou estimator anyway.

• @Hans I haven't followed the academic literature particularly closely over the past 5 years. However, based on my own personal messing around with financial data I'd say that it is hard to beat 5-minute realized variance, and for more liquid assets you can bring that down to 1 or 2-minute realized variance. These estimators have the bonus of being very simple to implement. I think these estimators perform well because trying to model autocorrelations (like realized kernels etc do) introduces a lot of additional estimation error due to the increased implied parameter space. Sep 11 '20 at 0:45
• @Hans I'm also rather fond of a procedure I designed during my PhD, described here. However, it is hard to see this procedure ever becoming popular in the main-stream academic literature since it is hard (impossible?) to do the asymptotic theory in the popular continuous-time semi-martingale plus noise framework. You need to use discrete-time modelling to justify the estimator theoretically and for whatever reason the academic literature tends to avoid that for high frequency data (but that is another story...) Sep 11 '20 at 0:47
• Thank you for the informative update. What about even higher frequency data than 1 minutes tick data? Does the same methodology apply? I do not quite understand your statement "I think these estimators perform well because trying to model autocorrelations (like realized kernels etc do) introduces a lot of additional estimation error due to the increased implied parameter space." You seem to say the estimators perform well BECAUSE they introduce more error. It seems paradoxical. What am I missing? I will look into your PhD thesis. Thank you very much!
– Hans
Sep 11 '20 at 20:04
• @Hans The higher the frequency of the financial returns, the more likely they will exhibit non-zero autocorrelation (typically negative autocorrelation due to microstructure effects like bid-ask bounce). Realized variance (of any frequency) assumes the autocorrelation is zero. Thus you should only use realized variance at frequencies where the zero autocorrelation is mostly satisfied (typically 5 mins, but maybe 1 or 2 mins for very liquid assets). Estimators like Realized kernels use every transaction. So by design they have to work with non-zero autocorrelation in the data (contd) Sep 13 '20 at 12:01
• @Hans All the ultra high frequency estimators (realized kernels, pre-average realized variance, mult-scale realized variance etc) have various estimation techniques for dealing with the autocorrelation. But you don't get something for nothing in statistics. By trying to estimate the autocorrelation (in some sense) you introduce estimation error. So there is a trade-off. Maybe the additional estimation error is small, so it is worth it to use all the data. But maybe the additional estimation error is large, in which case it is better to just use low freq realized variance - which I prefer. Sep 13 '20 at 12:05

If you calculate this estimator point by point, it could very well go negative. But the essence of this estimator is that statistically, it hovers around the realized variance, so it "estimates" the realized variance well. Even though the realized variance is by definition non-negative, the estimated distribution does not have to be.

In practice can you really use this estimator in place of real-time variance? It's like shooting a target through a blurred lens. You could miss by a lot.

A potential use case of this estimator though, is that if you only care about high variance moments in your trading algorithm. Negative values in this case become irrelevant. Zhou's estimator is easily calculated online as market data streams in.

I found this presentation by Jim Gatheral in 2006 on this subject. His conclusion is that Zhou's estimator outshined other estimators in the survey.

By the way I learned something from other top answers, especially the realized kernel method. It looks to me that Zhou's original paper set up the ground, and others generalized the problem into a kernel problem. Now it just becomes a study of different kernels.

I think it's ok: the total correction is $\approx -N\eta^2 \sim N^{-1/2},$ which tends to zero for large $N.$ So as long as it is a correction to the estimator $Q > 0$ you are ok.

• If I use this on high frequency stock data, sometimes the Zhou estimator yields a negative realized variance which means the volatility is imaginary. I think this is a problem? May 21 '15 at 18:42
• if $Q > 0$ and $N$ is large, $Q + C/N^{1/2} > 0,$ right? May 21 '15 at 19:09
• Hmm... what are you trying to say? That negative volatility is ok? Do you have any suggestions on how to handle the bias correcting term which can be negative? I am thinking of just using the absolute value, but that is not really good. May 21 '15 at 21:31
• I am saying, that the term you are worried about is not the whole volatility estimate, but rather a correction. And for large $N$ this term (even when negative) won't make your volatility estimate negative. May 21 '15 at 21:35
• Intraday Realized Variance for a whole day of High Frequency tick data, sometimes is negative for certain days. The correction term is too negative, yielding negative variance for the whole day. Do you think I have a bug in my implementation, or can it occur that intraday realized variance becomes negative for a whole day? We see that the realized variance can be negative, it is not a square. May 25 '15 at 22:56

Well for real hf stock tick data, i do get negativa volatility. So it can happens for large N

• Hi user16313, what do you mean by negative volatility, as using the common it must be positive? May 22 '15 at 7:36