Filling a few missing data in time series?

Question

I'm writing a paper about Uncertainty indices like VIX, etc. I already collected all data but it seems that some of the variables got a few or a little more missing data. I have daily and monthly data ready: https://cl.ly/8e1080296b31

What is the appropriate way to handle this case? It does not have to be very fancy as my econometrics background is basic.

If someone goes the extra 3 miles and fill the data will be even greater.

Answer 1

I would personally delete those days so you dont change the data distribution. If you really need to fill those blanks, random sample imputation would be the way to go.

Answer 2

I checked the VIX values that you report "missing" in 2007. They appear to be holidays (days when stocks don't trade and the VIX is not produced). For example 7/4/2007 is the Fourth of July holiday, 9/3/2007 is Memorial Day and so on.

A possible solution is to fill in on these days the VIX value for the previous day, since this is the "last known value". Another solution is to simply not include in your study any dates that are holidays. The NYSE publishes list of days on which the NYSE is closed for trading. For example the NYSE holidays of 2007 were: 20070101, 20070102, 20070115, 20070219, 20070406, 20070528, 20070704, 20070903, 20071122, 20071225.

Answer 3

You have two options here: the first is to discard any missing value and the second one is to impute(fill-in) all missing values.

Regarding the second approach the simplest way is to impute the missing values with the mean or the median of the non-missing values. A more sophisticated approach is to estimate a predictive model for the feature based on other features and then impute each missing value by the prediction of the model.

Keep in mind, that first you need to determine whether your data are missing at random (MAR). (for more info refer to MAR, MCAR). Most imputation methods rely on missing completely at random (MCAR)

Answer 4

There are two other possibilities not mentioned in the above answers.

Consider the time series $x_{t+1}=\beta{x}_t+\epsilon_{t+1}$. Assume $t\in\{81,82\}$ are missing. The question is why are they missing.

Consider three possible cases. The first is that it is a holiday or a similar day where there was no activity. The second is omission as a recording error. The third is hiding the data as it may be embarrassing to someone.

For the first two, it is simple. You alter your likelihood function in this one case. Noting for this example that $$x_{81}=\beta{x}_{80}+\epsilon_{81}$$ and $$x_{82}=\beta^2{x}_{80}+\beta\epsilon_{81}+\epsilon_{82}$$ and $$x_{83}=\beta^3x_{80}+\beta^2\epsilon_{81}+\beta\epsilon_{82}+\epsilon_{83}.$$

To make the example simple, let us assume that $\epsilon_t\sim\mathcal{N}(0,\sigma^2),\forall{t}$.

The simple case, where a holiday intervened is simple. You would note that no error could have happened on those days, so $\epsilon_{81}=\epsilon_{82}=0$.

You would update the posterior probability of $(\beta;\sigma^2)$ by simply cubing $\beta$ in the likelihood function. You can do this because $$\pi(\beta,\sigma^2|x_1\dots{x}_{80};x_{83})$$ follows from $$\pi(\beta,\sigma^2|x_1\dots{x}_{80})$$ via Bayes theorem where $$\pi(\beta,\sigma^2|x_1\dots{x}_{80})=\frac{\prod_{t=1}^{80}f(x_t|\beta;\sigma^2)\pi(\beta;\sigma^2)}{\int_0^\infty\int_{-\infty}^\infty\prod_{t=1}^{80}f(x_t|\beta;\sigma^2)\pi(\beta;\sigma^2)\mathrm{d}\sigma^2\mathrm{d}\beta}$$ and $$\pi(\beta,\sigma^2|x_1\dots{x}_{80};x_{83})=\frac{f(x_{83}|\beta;\sigma^2)\pi(\beta,\sigma^2|x_1\dots{x}_{80})}{\int_0^\infty\int_{-\infty}^\infty{f}(x_{83}|\beta;\sigma^2)\pi(\beta,\sigma^2|x_1\dots{x}_{80})\mathrm{d}\sigma^2\mathrm{d}\beta}.$$

In the case of an omission, there are two choices. The first is to treat it like a holiday because the alternative is to estimate the posterior of $\{\beta,\sigma^2,\epsilon_{81},\epsilon_{82}\}.$ It will end up with an answer that is no different than omitting them unless you have other variables which are correlated with $x_t$. In the case of at least one associated variable, you can improve the estimation of $\beta$ and $\sigma^2$ by estimating the missing variables as if they were parameters and then marginalizing them out via Bayes rule.

In the third case, where the omission is purposeful and assuming there is no correlated data, you should estimate the missing variable my inserting a prior density for them that estimates the size of the hidden variables.