# Significance Of Missing Data for RMSE Estimation

I have a time series covering ten years of daily close prices, which I compare to a theoretical time series generated by a model. The original series has a handful of missing data points (~2%), some of which occur in succession.

I intend to calculate a running RMSE between the two. Is it valid to use some basic interpolation technique, or is a more advanced ARIMA (or such) model necessary? Handling of missing data is not intended to be the main focus of the exercise, and is an issue I'm not so familiar with, but I don't want to brush over the problem if it has significant impact.

What about for more/less frequent occurrences of missing data?

## 2 Answers

Interpolation will create problems for you unless your work is entirely predictive. It will still be a problem, but only in understating the spread of your predictions.

The best solution to this problem is to solve it using Bayesian methods. This is due to how Bayesian methods handle missing data. If this is an academic exercise, you don't want to do it, but if it is a real-world problem, then you do.

Consider your options under a Frequentist model. If you want to preserve the center of location, then you should obviously do something similar to interpolation, but then you will probably understate the uncertainty in the model. With just 2% missing the impact is probably negligible, but with some time series, this is a large issue. Conversely, if you randomly generate an observation based on the parameter estimates for the rest of the series, you will probably preserve the uncertainty, but damage the predictive power. You could simply run the series from the point where there is no missing data, but then you discard information, which is never good. Finally, you could perform a meta-analysis. There are studies in the literature on performing the meta-analysis of time series. If you are to remain in the Frequentist camp, this is probably your best bet.

The Bayesian solution to this case depends in part as to why the data is missing in the first place. There are a number of potential reasons as to why data could be missing.

Imagine that your data was made up of accounts payable and in quarter 3 of 2008 the payables were zero so that rather than report zero the report is a missing entry. In that case, those values should be recorded as zeros, and your likelihood function should include the case where values may at times be zero. It could be due to censoring, and there is an entire branch of statistics involving censoring, and it would be too long for this post. If it is due to censoring, then you should be researching censored models. My guess, however, since the tone of your post seems to imply that the results are random ommissions the solution is rather simple.

Normal Bayesian regression takes the sample and treats it as a set of constants and assigns a probability distribution to the unobservable parameters. When some of the sample is missing, the missing points can be treated as an unobserved value, and a probability distribution can be assigned to each one of them. For example, if $x_{32}=25$ and $x_{34}=27$ then you could set a prior distribution for $x_{33}$ to get a posterior density for the value.

Bayesian methods, in the event you have not used them before, create probability distributions as answers rather than points. For example, instead of $\hat{\beta}=.075$ you may get a result such as $\beta\sim\mathcal{N}(.075,.05^2)$. It treats parameters as random numbers drawn from a distribution and observations as constants. This is the reverse logic to Frequentist methods. Samples are randomly drawn, and parameters are fixed in Frequentist methods. Parameters are not random, in the sense of chance, in Bayesian methods but in the sense that their true value is uncertain. Parameters are considered unobservables.

The distribution of answers depends upon what you know about the answer from information outside the sample itself. The distribution of the prior information is usually called "the prior." Just as meta-analysis requires subjective choices, the prior is a formalization of subjective choices. You include whatever information you have from prior research and professional knowledge.

Imagine you had a simple, stationary model of $x_{t+1}=\beta{x_t}+\alpha+\epsilon_{t+1}$. From other studies and professional experience you believe that with a high degree of probability $\beta\in[0,.2]$, $\alpha\approx{0}$, and the variability $\sigma\approx{1}$. If there were no missing data, you would create a tri-variate prior distribution to state approximately where the parameters are. To give an idea of how you would construct the marginal distribution of $\beta$, given that it is believed to be in the range $[0,.20]$ a simple choice for the distribution is $\beta\sim\mathcal{N}\left(.1,\left(\frac{.1}{3}\right)^2\right)$. You would use the empirical rule to set the variability in the range of possible values.

With one missing value, you would assign a probability distribution for, in this case, $x_{33}$. The center could be the interpolated value and the variability at least as large as the variability in the underlying data that was observed. Using this methodology has a number of advantages. First, Bayesian methods have a process called marginalization, that allows an analysis of the missing variable directly. By weak analogy to Frequentist methods, this allows the researcher a way to assign a p-value to the missing value. You can test to see if the resulting value is a reasonable value.

Imagine for this example $\Pr(x_{33}|x_1\dots{x_{32}},x_{34}\dots{x_N})\sim\mathcal{N}(37,.01^2)$. This is very far from where you believe it should be. This is a strong warning that something is very wrong with your overall model. For the math to place the missing value so very far away from where you believe it should be, implies that something else is going on if such an extreme value is necessary to make your model to make sense. If you are publishing your results, you should publish an analysis of the missing points.

This overall method is good in that it barely disturbs the location of $\alpha,\beta,$ or $\sigma$ from the population parameter. It allows you to disclose exactly what happened and it allows you to analyze the validity of your model.

A second option is to alter your likelihood function by noting that if $x_{t+1}=\beta{x_t}+\alpha+\epsilon_{t+1}$ and if one day is missing then by iteration $x_{t+1}=\beta\left[{\beta{x_{t-1}}+\alpha}\right]+\alpha+\epsilon_{t+1}$. There is no $\epsilon_t$ because no actual data happened. This allows you to simply ignore missing values. The danger of this is that if the omissions are not due to things like weekends or holidays, then there really was an observation that is, in a sense, being ignored; if it simply wasn't recorded.

The dual advantage of the Bayesian method is that it forces formal modeling of what is missing, and why it is missing; and a method to analyze that model and to disclose it for outside criticism. If you have not used Bayesian methods, there is a learning curve. The disadvantage is computational complexity and the learning curve.

When your solution is a p-dimensional probability density function instead of p points, then you have to have a lot more computation. This takes extra time, sometimes a lot of extra time. Furthermore, there is a lot more to analyze. If you believe that some paramter value is far from where it should be, then there is work to do. With the Frequentist model you are assured it is the best fit to the data and so anything unexpected need not be analyzed. With the Bayesian model, finding something unexpected should trigger an analysis. Getting a p-dimensional density and marginalizing it down to one dimension at a time takes work. Analyzing potential interactions among the dimensions takes more time.

The Bayesian method creates more information about the math of your model, but not necessarily more information about the data or the parameters.

Still, if I had omissions, I would use a Bayesian method. If I had no omissions, I would probably use OLS for the homoskedastic, stationary case. It is costly to generate all the additional work.

This is a pretty big topic, and whole books are written on missing data. Just to follow up on the previous comment, Bayesian methods are actually only one of the possible methods for dealing with missing data, and frequentists also have viable ways of dealing with it.

First, you should check out some easily available references, e.g., Wikipedia: Missing Data. The data deletion process/missing data mechanism is of importance. Is it Missing Completely at Random (MCAR) (says what it means) or just Missing at Random (poor nomenclature, but missingness may depend on variables in your dataset), or Not Missing at Random (e.g., censored data), where it depends entirely on the observation.

In the case you're dealing with it sounds like MCAR is not necessarily a bad assumption to make. If you're only doing OLS, then you basically only need to worry about estimating means and covariance matrices between your dependent and independent variables. In this literature, they talk about imputation (imputing the missing values) and inference (e.g., estimate the means and the covariance matrix, from the data that you have).

For the case of estimating the mean/covariance matrix, I have personally used the EM method, see for instance Schneider: Analysis of incomplete climate data: Estimation of mean values and covariance matrices and imputation of missing values, (you'll notice it's been cited some 590 times, which makes it a sort of minor classic in this area).

If it's more complicated, just search on those terms, and you'll find plenty of good material. A Great reference material on the topic is Little and Rubin: Statistical Analysis with Missing Data. Little and Rubin cover many different methods, both Frequentist and Bayesian. You'll find there are newer review articles, given the continuing development of literature in this area.

If your estimators are nonlinear and not just some simple OLS, then you should probably start with some google scholar searches, both for classics (lots of citations) and maybe some literature review like a "What do we know..." or "State of the Art..." type article, which will also have a good bibliography that could help sort you out.

BTW, some of these methods are so well covered that you can even find software that does the imputation and the inference directly. I believe both SAS and Stata have packages that handle this. There may even be some python repo.

There are of course review articles just covering Bayesian methods. AFAIK, Bayesian methods rely heavily on MCMC, while the multiple-imputation methods are usually an iterative method, where you just run until it appears to have converged. They're all numerical approximation methods, but MCMC answers change each time you run them (unless you save your pseudo-random number generation seed). This for me is a turn-off. Nowadays, you can use if you have some nice package like Stan that does MCMC efficiently and smooths the posterior distributions, it is probably one of the best ways to make the randomness of the answer seem more acceptable.