Mean Variance optimization on hourly data with gaps

Question

I'm building a mean variance optimizer for a portfolio of FX, commodity and bond futures. The input data is hourly returns for each underlying. Given each underlying has different market opening hours, this results in data with lots of null returns.

An example return table follows:

┌──────────────────┬─────────┬─────────┬───────────┐
│  DateTime (UTC)  │ EURUSD  │ SOYBNSD │ USB10YUSD │
├──────────────────┼─────────┼─────────┼───────────┤
│ 2005-03-02 07:00 │  0.0035 │     0.0 │       0.0 │
│ 2005-03-02 08:00 │ -0.0044 │     0.0 │       0.0 │
│ 2005-03-02 09:00 │  0.0019 │     0.0 │       0.0 │
│ 2005-03-02 10:00 │  0.0041 │  0.0082 │  -0.00002 │
│ 2005-03-02 11:00 │ -0.0053 │  0.0074 │   0.00003 │
└──────────────────┴─────────┴─────────┴───────────┘

There are probably a multitude of problems with this, including mis-representing volatility and correlation (in particular during periods that overlap with many market holidays).

I can think of several approaches to working around my particular scenario:

1. Do nothing.

   • This is what I've tried, and it seems to cause a headache for the optimizer as many rebalances simply end up with the initial guess (equal weight). I have yet to dig into exactly why this happens but my guess is that the correlation between the commodity and bond underlyings is greatly overstated and the volatility of the same understated.

2. Only keep rows where all assets have had returns.

   • Not a good idea because there may legitimately be days with zero return, and this also severely limits the data used for FX pairs that trade nearly 24/5.

3. For each underlying, convert returns during closed market hours to NaN.

   • For volatility this would make sense but calculating Pearson's correlation would be trickier. I can see on https://stats.stackexchange.com/a/7945 that a 1-dimensional cross-variogram is suggested, however I'm not sure this would work for my case as I'm dealing with "gap distances" that vary greatly (from 0 (during opening hours) to 16 hours (overnight) to many more hours (weekends/holidays)). Am I right in thinking this?

I have sought to find how this is dealt with in academia but found only examples of MVO on daily data, where it seems you typically cull the null rows and use only days where all assets have had returns.

Is anyone able to share insight into how they would tackle this problem?

Answer 1

Disclaimer If I were doing it, I would abandon mean-variance finance, however, if my employer insisted that I do it, then I would treat it as a modeled parameter using a Bayesian method.

Most Bayesian methods seek a posterior of the sort such as $$\Pr(\theta|\mathbf{x};\mathbf{y}),$$ where $\mathbf{x}$ and $\mathbf{y}$ are variables. Ignoring the fundamental differences of interpretation and calculation, this is no different than a Frequentist method. Take your data and construct your estimator for $\theta$.

In this case, though, imagine some of $\mathbf{x}$ did not exist because of a lack of overlap in trading times. People are still engaging in trades with their mental models intact as to what they think are the unrevealed prices. So let $\mathbf{x}_i$ be the set of observed cases and $\tilde{\mathbf{x}}$ be the cases where there are no observable variables. The pairing on the $y$ side is $\mathbf{y}_i$ and $\mathbf{y}_j$, where $j$ is the period of missing $x$'s.

Then, during the mutually observed period, I would calculate $$\Pr(\theta|\mathbf{x}_i,\mathbf{y}_i)$$ and use this relationship to form a prior distribution for the second case in order to solve $$\Pr(\theta;\tilde{\mathbf{x}}|\mathbf{y}_j).$$

Once I had $\Pr(\theta;\tilde{\mathbf{x}}|\mathbf{y}_j) $ I would marginalize out $\tilde{\mathbf{x}}$. Assuming differentiability, $$\Pr(\theta|\mathbf{x}_i,\mathbf{y})=\int_{\tilde{\mathbf{x}}\in\chi}\Pr(\theta;\tilde{\mathbf{x}}|\mathbf{y}_j)\mathrm{d}\tilde{\mathbf{x}}.$$

If you have not worked with Bayesian methods before, I treated the unobserved values as a parameter to be estimated and then removed the effect of that variable by integrating out the uncertainty in it. Instead of having one value for the data, you have at each missing data point a distribution of possible values that could have obtained at that time. Of course, a distribution is not what you want, but you make that distribution vanish by integrating it out of existence. Given the theoretical underpinnings of the CAPM, I would then find the posterior mean for each $\beta$ and that would be my model.

Because of the very high dimensionality of the model, in order to avoid marginalization paradoxes, you will want to put a very diffuse proper prior on each $\beta$ to guarantee it integrates to unity. Given the general acceptance in the field of LASSO or Ridge regression, I would put something on or near zero for the beta parameters with a normal density and a very diffuse gamma distribution for the scale parameters. That would allow you to calculate the initial portion with a conjugate prior, though that will not be possible for the second portion.

See, for example, https://www.faculty.agecon.vt.edu/AAEC5126/module6/BayesNormalConjugate.pdf

EDIT If you have never used a Bayesian method before, you should seek detailed support from someone you work with. I state this because your comment has the feel of someone who hasn't done it before.

A Bayesian prior summarizes all of your knowledge about the location of the parameters in a model, not just $\beta$. In this case, if $\tilde{\mathbf{x}}$ is being treated as a parameter, then it also needs a prior.

A prior is proper if $$\int_{\theta\in\Theta}\pi(\theta)\mathrm{d}\theta=1.$$ It is improper if that statement is not true. Frequentist solutions usually map to a Bayesian solution with an improper prior. The challenge is that for a normal likelihood, when you have three or more independent variables, then an improper prior will cause the posterior to not integrate to one. You will get paradoxes in your solution.

The prior comes from information outside the sample. You would assign a prior for each parameter on the full data set. The difference between Bayesian and Frequentist methods is that in Frequentist methods the sample is random. In Bayesian methods, the parameters are the random variables. The sample is treated as a constant. For the full set, you could assign a normal-gamma or normal-inverse-gamma distribution. For the partial data set, you would use the posterior of the full set as the prior of the new set, but add one dimension, your believed distribution for $\tilde{\mathbf{x}}.$

You are correct, you would capture your prior from the $\mathbf{x}_i$, but it wouldn't be a simulation. As you observed more and more data from outside the full set, your shape will change. Each observation will cause Bayesian updating which will change the distribution of $\mathbf{x}_i$.