Why quants think that the risk-neutral measure should not be used for financial forecasting?

Question

In posts regarding the $\mathbb{P}$ vs $\mathbb{Q}$ debate (see 1, 2, 3 or 4), most answers conclude that historical-based forecast are better suited than risk-neutral models for financial predictions.

However, this conclusion seems to be based on sentimental beliefs. In fact, empirical analyses comparing historical and risk-neutral predictions tend to conclude that risk-neutral models perform better than historical models. See, for instance, these recent papers: 1, 2 or these comprehensive surveys: 1 and 2.

While risk-neutral forecasts have theoretical flaws, a rational assessment would entail: (i) evaluating how these flaws actually impact future predictions and (ii) comparing the flaws observed in risk-neutral predictions with the biases of historical models.

However, since most people already seem to have a notably strong view, this pre-conditioning could be relegating what should be, conceptually, the core of the discussion (i.e.: performing empirical analyses and comparing forecast models with an agnostic view)

So, why it is that we adopt such pre-conditioned view regarding risk-neutral predictions?

Answer 1

There is a deeper issue. Frequentist distributions are not probability distributions because they are designed to be minimax distributions rather than actual distributions. This ignores all of the other problems and this also ignores risk-neutral versus any other measure of risk aversion.

An even deeper issue is that these models presume that the parameters are known. If you have a wealth model such as $w_{t+1}=Rw_t+\epsilon_{t+1}$, where $\epsilon_t\sim\mathcal{N}(0,\sigma^2_t)$, then if $R$ and $\sigma^2_t$ are known, then the mean-variance results follow. However, if the parameters are not known, then they can never be known.

See:

White, J. S. (1958). The limiting distribution of the serial correlation coefficient in the explosive case. The Annals of Mathematical Statistics, 29(4):1188–1197

This shocking proof, when combined with Mann and Wald's proof

Mann, H. and Wald, A. (1943). On the statistical treatment of linear stochastic difference equations. Econometrica, 11:173–200.

Imply that if mean-variance models are true, then no solution can be constructed using their axioms.

For a solution to the question you are asking, however, see

Harris, D.E.(2017) The Distribution of Returns. Journal of Mathematical Finance, 7, 769-804,

which derives the distributions of returns. You can use these as Bayesian likelihood functions and arrive at a coherent predictive distribution.

The Bayesian predictive distribution is: $$\Pr(\tilde{x}=k|\mathbf{X})=\int_{\theta\in\Theta}\Pr(\tilde{x}=k|\theta)\Pr(\theta|\mathbf{X})\mathrm{d}\theta,\forall{k}\in\chi,$$ where $\chi$ is the sample space, $\Theta$ is the parameter space, $\theta$ is a parameter or vector of parameters in $\Theta$, $\mathbf{X}$ is the observed sample and $\tilde{x}$ is some predicted value.

Notice that $\Pr(\tilde{x}|\mathbf{X})$ does not depend upon the values for $\theta$ as any uncertainty as to its true value has been marginalized out so that the prediction only depends upon the information that you observed.

The predictive distribution of $\tilde{x}$ is coherent so that a bookie, or market-maker, offering prices or odds based on the information in $\mathbf{X}$ cannot be gamed and take a sure loss by a clever actor or set of actors. That is never true for Frequentist solutions. You can rig a game based on a Frequentist statistic if you are crafty enough.

Whereas the Frequentist model assumes the parameters are known, the Bayesian model ends with a result that they do not matter for the purposes of making a prediction. That is a powerful result. The former is fragile, the latter is robust.

As to why we adopt these strong views, it is because we are people. Even economists who do not work for the Street have vested interests. It is challenging to be a scientist when your salary is tied to your outcomes.

Answer 2

In their book "Counterparty Credit Risk, Collateral and Funding" D. Brigo, M. Morini and A. Pallavicini start with a dialogue between a Physics PhD graduate and an experienced practitioner of Quantitative Finance.

The topic of P vs Q is presented in that dialogue in a manner meant to be understandable to a new comer. I would certainly recommend you to have a look!

This dialogue is also available on the arXiv.

I am also adding a couple quotes from that dialogue:

.. "Q: Ok. We simulate under P because we want the risk statistics of the portfolio in the real world, under the physical probability measure, and not under the so called pricing measure Q" ..

.. "Q: And how is this Q related to P? [still puzzled] A: The two measures are related by a mathematical relationship that de- pends on risk aversion, or market price of risk. In the simplest models the real expected rate of return is given by the risk free rate plus the market price of risk times the volatility. Indeed the ”expected” return of an asset depends on the probability measure that is used. For exam- ple, under P the average rate of return of an asset is hard to estimate, whereas under Q one knows that the rate of return will be the risk free rate, since dependence on the real rate of return can be hedged away through replication techniques. [starts looking tired]"

Answer 3

Perhaps a case of views based upon theoretical possibilities rather than empirical realities?

In theory, $P$ and $Q$ can be extremely different

• $P$ is the real world, actual probability measure.
• $Q$ is an artificial probability measure constructed so that security prices are risk neutral expectations (discounted at risk free rate) under measure $Q$.

The connection between the two is through state prices. If state prices are constant, then $P$ and $Q$ are the same. But with arbitrary state prices, $P$ and $Q$ can be made arbitrarily different from each other. The probability of a hurricane under the risk neutral measure can deviate arbitrarily from the probability of a hurricane under the true probability measure depending on the market price of hurricane risk. If risk prices are extreme enough, a 1% probability event may have 20% probability under the risk neutral measure. (Economic theory may have some objection to this, but the basic math of risk neutral pricing does not.)

Quick review: the pricing kernel connects $P$ and $Q$

Let $p(X)$ be an asset pricing function that returns the price of payoff $X$. If $p$ is linear, then it can be written as an inner product of the payoff $X$ with a pricing kernel $\pi$. $$p(X) = \mathbb{E}^P\left[ \pi X \right]$$

Then define Radon-Nikodym derivative $\frac{dQ}{dP} = \pi R_f$ and conduct a change of measure. $$ p(X) = \frac{1}{R_f}\mathbb{E}^P\left[\frac{dQ}{dP}X \right] = \frac{1}{R_f} \mathbb{E}^Q[X] $$

So depending what pricing kernel $\pi$ is, some notion of distance between $P$ and $Q$ could be quite large (since the linear $p$ assumption on its own doesn't put meaningful constraints on $\pi$).

Answer 4

The risk neutral density is a mathematical trick to allow pricing of options. As it has little bearing on reality, it makes little sense to simulate from it for the purposes of forecasting real results.

I would argue that any paper that shows that the risk neutral density forecasts better than the real world is either suffering from a specification or an estimation problem.

Perhaps the best empirical study that I have seen on the topic is the following paper, which examines the real world density, the risk neutral density, and the link between the two for the (very general) CGMY model:

http://finance.martinsewell.com/stylized-facts/distribution/CarrGemanMadanYor2002.pdf

Answer 5

I'm not sure I'd call it 'subjective' or 'pre-conditioned'. Traditionally, ensuring the absence of arbitrage is a guiding principle for pricing, while the risk-neutral measure is most frequently used for theoretical results. Assuming you'll be using your forecasting for trading activities, using AoA to determine price is the right way to go.

Consider a simple case of a forward's expected value being the spot price compounded at the current 'risk-free rate'. I'm not sure how you'd argue pricing/forecasting a forward using the risk-free rate was appropriate in anything other than a theoretical risk-neutral world. Same goes for pretty much any asset for that matter. I'd be happy to take the other side of the trade if you felt differently though.