What volatility estimator for continuous data and small time window?

Question

I want to know which volatility estimator should I use for the following scenario:

I am implementing a market making bot and therefore I need to make estimations of the volatility of the price in the fashion of asking: What was the volatility of the price in the last couple of Minutes? (5-30 Minutes)

The data I've got available for the estimation is a set of all the prices in that time period at an interval of about 2-5 seconds, which are about 500 data point. And every time a new data point is added to the set, all data points that are older than the period of interest (5-30 Minutes) are removed from it.

Right now I use a basic estimator that calculates the variance of all the prices, however the problem with that is: the volatility oscillates way to much. I would expect the volatility to change slow and continuous over time.

Answer 1

First, you should use an exponential moving average, since the amount of state you need to keep is much smaller than for a simple moving average.

Second the well known estimator of volatility,

$$ \hat{\sigma} = \sqrt{\frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2} $$

is not very robust, since the squaring amplifies the contribution of outliers (which is why you are observing a very noisy volatility estimate - high frequency data has a lot of outliers).

Instead, consider using a mean absolute deviation estimate,

$$ \hat{\sigma}_{MAD} = \frac{1}{n} \sum_{i=1}^n |x_i - \bar{x}| $$

which is more robust to outliers. You need to multiply this by a factor of $\sqrt{\pi/2}$ so that it matches the scale of the standard deviation estimator above.

The quantities $x_i$ should be the price differences from tick to tick, i.e.

$$ x_i = p_i - p_{i-1} $$

or maybe the returns,

$$ x_i = \frac{p_i}{p_{i-1}} - 1 $$

You might want to consider thresholding the $x_i$ to some maximum value, say 5x or 10x the current volatility estimate, to reduce the impact of outliers even further.