Online algorithm for selecting smoothing parameter?

In Online Algorithms in High-frequency Trading the authors demonstrate online, exponentially-weighted algorithms for mean, variance, and linear regression.

The authors estimate their smoothing parameter $\alpha$ in-sample, but state "Another approach would be to estimate the optimal alpha online as well."

What would be an example of how to estimate the smoothing parameter online?

First of all, I do not believe the "optimal smoothing" of an estimator (like the mean or the variance) and the "regression case" are the same.

The smoothing of an existing estimator (like mean or variance in the blog post), is an univariate problem, where the regression is a multivariate one. In the regression case, you should be able to change the coefficients of the whole formula when you adjust the smoothing, but in the "simple" smoothing one you are not expected to change the formula.

I will answer to the first case here. For the regression case, you should have a look at ARIMAX models and Kalman filters, it is (according to me) difficult here to split in two the process of (1) obtaining the good model, (2) apply a smoothing.

Obtaining an adequate filtering window size once you have a formula (say the formula of the mean), is already difficult. Say you observe $X_1,X_2,\ldots,X_k,\ldots$ at time $\tau_1,\tau_2,\ldots,\tau_k,\ldots$. You have to go back to the formula of the mean, where does it come from?

It is easy to see that the usual mean is the solution to the following minimization problem:

$$\min_m \mathbb{E}(m-X)^2.$$

It minimizes the expected variance between the observed random variable and the "mean". Under simple assumptions ($X$ is i.i.d.) $m$ is the empirical mean, using as much data as possible. In terms of smoothing it would mean you take all the points since the start of the day (I uses your intraday example), and averages more data with time:

$$m_\infty(X_k,k\leq K)=\frac{1}{K} \sum_{k\leq K} X_k.$$

And the variance of this estimator is proportional to $\sqrt{\mathbb{V}(X)/K}$.

Why would you need something different?

If you want to take less points, for instance a sliding window of size $W$:

$$m_W(X_k,k\leq K)=\frac{1}{W} \sum_{k= K-W+1}^K X_k,$$

you will obtain again an unbiased estimator, but with a largest variance, since you replace $\sqrt{\mathbb{V}(X)/K}$, where $K$ goes to infinity with time, by $\sqrt{\mathbb{V}(X)/W}$, that is constant.

If you do that it is because you believe $X$ is not i.i.d. (independent and identically distributed). In such a case you can have $X$ being in fact drawn from a random variable until time $T$, and then it switches to another distribution.

It means that you need to detects this change, or at least to be not too much sensitive to it. If you take an exponential average you are in the second case: you do not try to know anything about the switches, you just implement a formula such that after a while you do not take the past into account at all, and you weight more the recent past that an older one.

But you can do better than that. And there is a large literature on the kind of problem, called **change detection*. Here is a good review: State-of-the-Art in Sequential Change-Point Detection. Shiryaev wrote a lot of good papers on this topic.

The generic methodology is to choose a contrast (i.e. a criterion that will be low when the state remains the same, and high during the change), and to threshold it to detect a change of regime. Then it is enough to reinit your smoothing. And practically to be confortable you can use an exponential moving average between two switches if you want.

For the mean, a typical and simple contrast is the recent empirical mean vs. an old one normalized by a standard deviation:

$$C(K):=\left|\frac{{\rm mean}(X_{K}:X_{K-h})-{\rm mean}(X_{K-2h}:X_{K-h})}{{\rm std}(X_{K-2h}:X_{K-h})}\right|,$$

with the notation $X_{a}:X_{b}$ the time series from $b$ to $a$.