How to build a regime-switching model which knows its own limits?

Question

In recent months I've come to the conclusion that there are not only certain regimes in the markets (like bear or bull) but phases where all models fail because we are in uncharted territory. The former are pockets of predictability, as I like to call them, the latter are phases where it is best to stay out of the markets altogether.

Another observation is that there seem to be variables that in and of themselves have very little forecasting power but seem to be useful in differentiating between different regimes. I haven't tested that rigorously but an idea why this could be the case would be that the relationship is highly non-linear, but there nevertheless.

As a very crude example lets take the VIX as the so called barometer of fear. It doesn't seem to be that good at forecasting returns, yet it seems that different levels show different regimes, i.e. a certain tendency in the market (low -> bull, high -> bear).

But when we have extreme readings the swings are extreme too, i.e. markets falling like a stone but sometimes very pronounced swing-backs too. That would be an example of complete unpredictability. There could also be a region between a real "low" and a real "high" reading where things are unpredictable too (even in probabilistic terms).

As another, more elaborate example take a look at this quant trading model from UBS

My question
How would you proceed in building a model that identifies different regimes in the data and meta-models its own limits? Which mathematical ansatz (approach) would you choose? How would you find the brackets (barriers/limits)? How would you test it?

Answer 1

High VIX arguably leads to less predictability of the market factor (i.e. market timing), but high volatility does lead to greater predictability of the cross-section of returns. Indeed, linear risk factor models have higher explanatory power during bear markets.

However, your goal is to build a better market timing model where the forecasts (and perhaps confidence levels) adjust to prevailing conditions.

I would take a look at Linear Quadratic Regulators - also known as state-feedback controllers. Your analogy of "meta-model" matches the idea of "controller" in LQR systems. These LQR systems are best when you have a set of dynamics or differential equations that describe a system.

These systems are popular in engineering and aerospace applications where you describe the evolution of a system in terms of physical laws such as the heat equation or fluid dynamics.

But you can use this in finance if you start with some linear model to predict the market factor and then identify a tool to best estimate the parameters for your model (maximum likelihood, Newton's method, etc.).

A close cousin of LQR are partially observable Markov decision processes (POMDPs) which is also worth exploring. (I am actually exploring these models myself so we will have to trade findings!)

Another approach to consider would be a Bayesian approach. When VIX is high, you shrink towards a prior (a cash position, or benchmark). When VIX is low you reduce the range of outcomes. A practical way to make this come together is Meucci's paper on entropy-pooling. When you have a forecast in a high VIX state you can assign a high variance to your prediction to reduce your confidence in the estimate. Meucci also has fully commented code for implementing entropy-pooling on his www.symmys.com website.

Note that both LQR and POMDPs are special cases of state-space models.

Answer 2

I'd use a Hidden Markov Model. Develop a statistical model of how the variables behave in different regime and establish transition probabilities between the regimes. Solve with the Viterbi Algorithm.