# How to estimate real-world probabilities

In the world of finance, Risk-neutral pricing allow us to estimate the fair value of derivatives using the risk free rate as the expected return of the underlyings.

However, the behavior of financial assets in the real-world might be substantially different to the evolution used in a risk-neutral context.

For instance, if I want to estimate the real-world probability of an equity asset reaching certain thresholds, which models and calibration techniques could be used?

In particular, some questions that may arise in the estimation of real-world probabilities are:

• Calibration: Should real-world probabilities be calibrated to current market prices or, alternative, historical data should be used for this type of estimation?
• No-arbitrage conditions: Could they be relaxed or they still play a role in the assessment of real-world probabilities?
• Expected returns: Assuming that I have already estimated the expected return of an asset $\mu$, how accurate would be a real-world estimation that combines a widely used evolution model (e.g. Geometric Brownian motion), with the use of $\mu$ instead of the risk free rate $r$?

Per comments, I understand that in order to estimate real-world probabilities:

• I should use expected returns instead of the risk-free rate.
• The asset evolution should still respect the no-arbitrage conditions (i.e: the real-world dynamics should still reproduce the current prices of vanilla options).

However, if we just use $\mu$ instead of $r$, the underlying asset behavior might not be consistent with the observed option prices. For instance, if we just change $r$ by $\mu$ (with $\mu>r$) the underlying asset dynamics will lead to call prices above its current market price, and put prices below its market price.

Therefore, in addition to use expected returns, which other adjustment might be needed in order to estimate real-world probabilities?

Any papers or references regarding real-world estimation will be greatly appreciated.

The risk-neutral measure $$\mathbb{Q}$$ is a mathematical construct which stems from the law of one price, also known as the principle of no riskless arbitrage and which you may already have heard of in the following terms: "there is no free lunch in financial markets".

This law is at the heart of securities' relative valuation, see this very nice paper by Emmanuel Derman ("Metaphors, Models & Theories", 2011) and some part of this discussion.

In what follows, assume for the sake of simplicity

• existence of a risk-free asset ;
• deterministic and constant rates, with risk-free rate $$r$$ ;
• no dividends and no additional equity funding costs.

How to relate $$\mathbb{Q}$$ to $$\mathbb{P}$$: some useful concepts

The risk-neutral measure $$\mathbb{Q}$$ is a probability measure which is equivalent to $$\mathbb{P}$$ and under which the prices of assets (I should rather say the price of self-financing portfolios composed of marketed securities to be perfectly rigorous), discounted at the risk-free rate, turn out to be martingales.

1. If one assumes there is no free lunch in the real world (hence under $$\mathbb{P}$$), then the above definition (more specifically the "equivalent" part) suggests that there will be no free lunch under $$\mathbb{Q}$$ either. To convince yourself have a look at the accepted answer to this SE question. This answers your question concerning no arbitrage conditions.

2. The martingale property is convenient since it allows us to represent asset prices as expectations conditional on the information we currenty have, which seems intuitive and natural. Indeed from the definition if $$X_t$$ is a $$\mathbb{Q}$$-martingale then $$X_0 = E^{\mathbb{Q}}[X_t \vert \mathcal{F}_0]$$

3. The adjective risk-neutral comes from the fact that, using a replication argument (static for linear contracts, dynamic for most of the others) and under the assumption of no free lunch (+ market completeness, continuous trading, no frictions), one can show that the true performance of the stock simply disappears from the option valuation problem. Risk aversion thus disappears and only the risk-free rate $$r$$ remains. This is exactly what Black-Scholes-Merton showed and which earned them the Nobel prize in the first place, see below.

A simple example: the Black-Scholes model

Assume that the stock price $$S_t$$ follows a GBM under $$\mathbb{P}$$ $$\frac{dS_t}{S_t} = \mu dt + \sigma dW_t^{\mathbb{P}}\ \ \ (1)$$ where $$\mu$$ is the expected performance of the stock and $$\sigma$$ the annualised volatility of log-returns. This equation describes the dynamics of the stock in the real world.

Consider the pricing (we are still under in the real world) of a contingent claim $$V_t = V(t,S_t)$$ of which the only thing we know is that it pays out $$\phi(S_T)$$ to its holder when $$t=T$$ (generic European option). Now, consider the following self-financing portfolio:

$$\Pi_t = V_t - \alpha S_t$$

Using Itô's lemma along with the self-financing property yields: \begin{align} d\Pi_t &= dV_t - \alpha dS_t \\ &= \left( \frac{\partial V}{\partial t} + \frac{1}{2} \sigma^2 S_t^2 \frac{\partial^2 V}{\partial S^2}\right) dt + \left( \frac{\partial V}{\partial S} - \alpha \right) dS_t\\ \end{align}

The original argument of Black-Scholes-Merton is then that, if we can dynamically rebalance the portfolio $$\Pi_t$$ so that the number of shares held is continuously adjusted to be equal to $$\alpha = \frac{\partial V}{\partial S}$$, then the portfolio $$\Pi_t$$ would drift at a deterministic rate which, by absence of arbitrage opportunity, should match the risk-free rate.

Writing this as $$d\Pi_t = \Pi_t r dt$$ and remembering that we've picked $$\alpha = \frac{\partial V}{\partial S}$$ to reach this conclusion, we have

\begin{align} &d\Pi_t = \Pi_t r dt \\ \iff& \left( \frac{\partial V}{\partial t} + \frac{1}{2} \sigma^2 S_t^2 \frac{\partial^2 V}{\partial S^2} \sigma^2 \right) dt = \left( V_t - \frac{\partial V}{\partial S} S_t \right) r dt \\ \iff& \frac{\partial V}{\partial t}(t,S) + r S \frac{\partial V}{\partial S}(t,S) + \frac{1}{2} \sigma^2 S^2 \frac{\partial^2 V}{\partial S^2}(t,S) - rV(t,S) = 0 \end{align} which is the famous Black-Scholes pricing equation. Now, the Feynmann-Kac theorem tells us that the solution to the above PDE can be computed as: $$V_0 = E^\mathbb{Q}[ e^{-rT} \phi(S_T) \vert \mathcal{F}_0 ]$$ where under a certain measure $$\mathbb{Q}$$ $$\frac{dS_t}{S_t} = r dt + \sigma dW_t^{\mathbb{Q}}$$ which shows that $$\frac{V_t}{B_t} \text{ and } \frac{S_t}{B_t} \text{ are } \mathbb{Q}\text{-martingales}$$ with $$B_t = e^{rt}$$ representing the value of the risk-free asset we mentioned in the introduction. Notice how $$\mu$$ has completely disappeared from the pricing equation.

Because this Feynman-Kac formula very much resembles a magical trick, let us zoom in on the change of measure from a more mathematical perspective (the above was indeed the financial argument... at least for deriving the pricing equation, not for expressing its solution in martingale form).

Starting from $$(1)$$, let us define the quantity $$\lambda$$ as the excess-return over the risk-free rate of our stock, expressed in volatility units (ie its Sharpe ratio): $$\lambda = \frac{\mu - r}{\sigma}$$ Plugging this into $$(1)$$ gives: $$\frac{dS_t}{S_t} = r dt + \sigma (dW_t^{\mathbb{P}} + \lambda dt)$$ Now Girsanov theorem tells us that if we define the Radon-Nikodym of the change of measure as $$\left. \frac{d\mathbb{Q}}{d\mathbb{P}} \right\vert_{\mathcal{F}_t} = \mathcal{E}(-\lambda W_t^{\mathbb{P}})$$ then the process $$W_t^{\mathbb{Q}} := W_t^{\mathbb{P}} - \langle W^{\mathbb{P}}, -\lambda W^{\mathbb{P}} \rangle_t = W_t^{\mathbb{P}} + \lambda t$$ will emerge as a $$\mathbb{Q}$$-Brownian motion, hence we can write: $$\frac{dS_t}{S_t} = r dt + \sigma dW_t^{\mathbb{Q}}$$

Okay, this might seem even more magic to you than earlier, but there is a rigorous mathematical treatment behind don't worry.

Anyway, an interesting feature of writing and manipulating the Radon-Nikodym derivative is that one can eventually show that:

$$V_0 = E^{\mathbb{Q}} \left[ \left. \frac{V_T}{B_T} \right\vert \mathcal{F}_0 \right] = E^{\mathbb{P}} \left[ \left. \frac{V_T}{B_T} \mathcal{E}(-\lambda W_T^\mathbb{P}) \right\vert \mathcal{F}_0 \right]$$

where I have used the Bayes' rule for condition expectations, with $$X := V_T/B_T,\ \ \ f := \frac{d\mathbb{Q}}{d\mathbb{P}} \vert \mathcal {F}_T = \mathcal{E}(-\lambda W_T^{\mathbb{P}}),\ \ \ E^\mathbb{P}[f \vert \mathcal{F}_0 ] = 1$$

The above result is extremely interesting and can here be re-expressed as

$$V_0 = E^{\mathbb{Q}} \left[ e^{-rT} \phi(S_T) \vert \mathcal{F}_0 \right] = E^{\mathbb{P}} \left[ e^{-\left(r+\frac{\lambda^2}{2}+\frac{\lambda}{T} W_T^{\mathbb{P}}\right)T} \phi(S_T) \vert \mathcal{F}_0 \right]$$

This shows that, under BS assumptions:

• The option price can be calculated as an expectation under $$\mathbb{Q}$$ in which case we discount cash flows at the risk-free rate.
• The option price can also be calculated as an expectation under $$\mathbb{P}$$ but this time we need to discount cash flows based on our risk-aversion, which transpires through the market risk premium $$\lambda$$ (which depends on $$\mu$$).

Therefore, in addition to use expected returns, which other adjustment might be needed in order to estimate real-world probabilities?

You need to use a stochastic discount factor accounting for the risk aversion, see above and further remarks below.

Estimating real-world probabilities assuming BS

You have different possibilities here. The first idea which springs to mind is to calibrate your diffusion model to observed time series. When doing that, you hope to get an estimate for $$\mu$$ and $$\sigma$$ in the GBM case. Now given what we just said earlier, you must be very careful when pricing under $$\mathbb{P}$$: you cannot discount at the risk-free rate. Also obtaining a statistically significant estimation for $$\mu$$ (and the latent equity risk premium) may not be as easy as it seems see the discussion here

It's more complicated than that when you choose another model than BS

The relationship: $$V_0 = E^{\mathbb{Q}} \left[ \frac{V_T}{B_T} \vert \mathcal{F}_0 \right] = E^{\mathbb{P}} \left[ \frac{V_T}{B_T} f \vert \mathcal{F}_0 \right]$$ with $$f = \left. \frac{d\mathbb{Q}}{d\mathbb{P}} \right\vert_{\mathcal{F}_T}$$ will hold (under mild technical conditions).

Compared to the risk-free discount factor $$DF (0,T):=1/B_T$$ the quantity $$SDF (0,T):=f/B_T$$ is best known as a Stochastic Discount Factor (maybe you've already heard about SDF models, this is precisely that) and we can write, without loss of generality:

$$V_0 = E^{\mathbb{Q}} \left[ DF (0,T) V_T \vert \mathcal{F}_0 \right] = E^{\mathbb{P}} \left[ SDF (0,T) V_T \vert \mathcal{F}_0 \right]$$

The problem is that, depending on the model assumptions you use, you cannot always have a simple and/or unique form for $$f$$ (hence $$SDF (0,T)$$) as it used to be the case in BS.

This is notably the case for incomplete models (i.e. models that include jumps and/or stochastic volatility etc.). So now you understand why when we need models to price options, we directly calibrate them under $$\mathbb{Q}$$ and not on time series observed under $$\mathbb{P}$$.

• @sets does that answer your question? – Quantuple May 24 '16 at 20:23
• Indeed. This is the type of answer that I was looking for. Are you aware of any book or paper with numerical examples on SDFs to develop some intuition on its practical implementation? – sets May 25 '16 at 6:59
• Also there seems plenty to related questions on SE, just use the "stochastic discount factor" keywords, see here for instance: quant.stackexchange.com/questions/15674/…. – Quantuple May 25 '16 at 8:37
• Well just use the equation I mentioned in the BS framework, along with fixed values of $\mu$, $\lambda$, $\sigma$ and $T$. You can compute both expectations with a simple Monte Carlo simulation for instance. – Quantuple May 25 '16 at 9:24
• @Hans good point. I've edited the link and posted the title of the paper. Thank you – Quantuple Jul 26 '19 at 21:40

You may want to consider splitting two important, yet very different concepts:

Pricing a derivative security with contingent payoff and forecasting an asset.

• Pricing a derivative can be achieved through setting up a hedge portfolio and track its evolution and "value" at any point in time before the derivative security pays off. Risk-neutral pricing is a handy tool to accomplish that. In most all cases do you need to possess knowledge of the underlying price dynamics which most likely depend on one or more random components, such as Brownian Motion.

• Estimating the probability of a non-contingent asset (such as a stock) reaching certain thresholds can be done entirely without the construct of any risk neutral probability measure. All you need is a pricing model and a parameter set (which you could estimate or derive from a fit to historical data) and run a simple Monte Carlo simulation. No need for risk neutral probabilities at all.

My point is that the concept of risk neutral pricing is not necessary if you want to estimate the probability of an asset with non contingent payoff to reach certain price levels. Your question was which models could be used to estimate the probability of reaching such thresholds: You can setup a pricing model, whose parameters you fit to past data, and throw it into a MC pricer. Check how many of the paths reach your thresholds and derive your probability. That is an example where you use real-world parameters to estimate a real-world probability.

EDIT (in response to edited question)

Calibration -> Calibrate real-world probabilities to historical data and models, incorporating risk-neutral probabilities, to current market prices.

No-arbitrage conditions: No they cannot be relaxed, and why would you want to do that? You look for a self-consistent model and if you calibrate to current market prices but throw overboard no-arbitrage conditions then you end up with incorrect probabilities because your model is distorted.

Historical fit: You can calibrate any model that incorporates real-world probabilities to historical data. Whether history repeats itself and whether your assumed risk premiums lead to the correct probabilities is an entirely different question.

Expected Returns: You do not have the choice when using real-probabilities; you cannot use the risk-free rate because in your world of real probabilities investors are risk averse and apply different utility curves, hence, you need to estimate risk premiums and expected returns instead of simply using a risk-free rate. As this is a pretty damning exercise it is the precise reason why risk-neutral probabilistic models are so attractive.

In short? A non risk-neutral probability model.

• I am afraid that this - while of course true - doesn't really address the question of the OP: "If I want to estimate the "real world" probability of an asset reaching certain thresholds, which models and alternatives could be used?" So I guess it would be helpful to expand your comment "All you need is a pricing model and a parameter set (which you could estimate or derive from a fit to historical data) and run a simple Monte Carlo simulation." – vonjd Jun 20 '13 at 17:08
• Fair, though I wanted to stress that risk neutral probabilities and estimating the probability of an asset with non-contingent payoff reaching certain thresholds are two very different and unrelated exercises. But will try to elaborate. Thanks – Matthias Wolf Jun 20 '13 at 17:14
• @MattWolf, I have just updated the question. Will be very helpful if you could elaborate your answer. – sets Jun 27 '13 at 11:43
• @sets, I edited my answer to reflect your edited question – Matthias Wolf Jun 28 '13 at 4:42

First of all, I must say that it's a very general question, and the answer can vary depending on type of assets you model.

In quant finance real world probabilities are generally used for risk management. It can be said, that in order to use real-world probabilities you have to calibrate your models to history. In order to obtain risk-neutral probabilities, you fit to market.

Simplest example - brownian motion for asset price. It is $\frac{dS}{S} = \mu dt + \sigma dW_t$ in real world and $\frac{dS}{S} = r dt + \sigma dW_t$ in risk-neutral world.

Where would you take $\mu$ from? The easiest way is to take history and estimate historical asset drift, or just calculate $\frac{1}{N}\sum_{i=1}^N\frac{S_{i+1}-S_i}{S_i}$.

Where would you take $r$ from? You just take current risk-free rate.

At the same time I must stress that quant finance models are IMHO unsuitable for long-term forecasting. In this case you have to seek for appropriate econometric model.

This is indeed one of the most difficult tasks to do (if not next to impossible).

I would say the standard reference is the following:
Expected Returns: An Investor's Guide to Harvesting Market Rewards by Antti Ilmanen

An abridged (but still about 170 pages long), yet more current - and free (!) version in different formats (pdf, mobi for the Kindle and epub) can be found here:
Expected Returns on Major Asset Classes by Antti Ilmanen

Addendum: A 8-page long summary of the main points can be found here:
Understanding Expected Returns by Antti Ilmanen

Two remarkably simple solutions have been missed. Let us go a completely different route. Let's assume that the standard models don't work sufficiently well, for whatever reason, and that we need a solution that is both model independent and either reveals long run frequencies or the probabilities given the data.

One method is fancy and so would look good on paper. The other is primitive and sounds boring, but satisfies the coherence principle. The first one is kernel density estimation, the second is an extension of a Bayesian multinomial estimation.

The advantage of the first is that it can provide a smooth curve as long as the bins are wide enough. The advantage of the second is that because it satisfies the coherence principle then gambles can be made from the estimates.

It is pretty easy to find decent work on estimating long-run frequencies using kernel density estimates. Between software, online books, paper books and journal articles there are a plethora of options.

You are likely less familiar with using Bayesian multinomial estimates to construct a density estimate. It is really just very fancy binning with the added property that you can use it to gamble.

First, I would transform my raw data to possible values of a $S_{t+1}$, the possible stock prices, and $K$, the strike price, so that I was reframing the problem as a return-style problem as $$\frac{S_{t+1}}{K}.$$ As you can always multiply by $K$ as it is a constant, there is no information loss. Conversely, unless you believe the distribution's form is a function of $K$ itself, the conversion would allow direct comparison of five dollar strike prices and fifty dollar strike prices. My first bin would be from zero to the first partition value and my last partition would be from $K$ to $\infty$.

Using this tool has two giant advantages. If the heavy-tailed distribution has no mean then the put will still have an expected return because only the distribution from zero to $K$ matters in terms of creating a division, while $K$ to $\infty$ only matters to complete the set, but not to set the price.

Each partition would have a parameter $\theta_i$ that is the estimator for the probability that a return will fall inside that partition. Unlike the maximum likelihood estimate or the minimum variance unbiased estimator, the Bayesian method produces a distribution of possible values for $\theta_i$ for each partition instead of a point estimator. Furthermore, $\theta_i$ could depend on data from adjacent partitions.

A prior distribution would need to be determined and a Dirichlet prior density function set based on what you believed to be the likely values in the bins. The likelihood function would just be the multinomial likelihood and the posterior would just be the multivariate Polya distribution. This will allow you to form the Bayesian posterior predictive density estimate. This density estimate can have fair gambles placed on it as it is both admissible and coherent.

This second method provides valid probabilities. It also allows you to consider the impact of mergers and bankruptcies in a single distribution or as the weighted average of the distributions. Its value is that it does not at all depend upon theory.

I have written a paper on how to do this and an empirical test, but rather than cite it, the subsection on the handling of the distribution is reproduced here in a more general form.

As to your specifics on calibration, expected values and no-arbitrage rules I felt I should comment. As to calibration, it is intrinsically calibrated because Bayesian methods estimate the probability a model is valid given the data rather than the probability of seeing the data given a model is true. Frequentist models need to be calibrated. Bayesian models do not assume a model is the true model and do not have this headache. This particular tool, less so, because there are no distributional assumptions present.

Expected values only exist in mesokurtic and platykurtic models, generally, and only sometimes in leptokurtic models. You can construct expected losses, given a loss happens, times the probability of loss. That is not an expected value, but it always exists. I built an option pricing model around it. It works. You do not even need expectations.

As to the absence of arbitrage opportunities, de Finetti's coherence principle is a weaker assumption than an absence of arbitrage opportunities. It merely assumes the market maker cannot be gamed in all states of nature, that is the market maker cannot be tricked into taking a sure loss, regardless of the outcomes. In essence, it says that if an arbitrage opportunity does form, then the market maker will take it. It does not require that no opportunities ever exist, only that it cannot be used against the market maker to produce income.

I used this method as a back-up because I separately wrote a paper deriving the distribution of returns under a very wide set of possible assumptions. It didn't just cover stocks, but things like antiques at Sotheby's, accounting ratios and single period discount bonds. Most of the distributions lack a first moment and so you cannot use standard option theory, or lack an analytic solution and so you cannot use a parametric solution. As most actual distributions are mixture distributions, at least in the paper, this method is a reasonable shortcut. This method also doesn't require anyone answering the question to be right as to the solution to option pricing, as it only requires the data to be true data. We could all be wrong, but the solution would be one that a valid gamble can be placed on.

You can definitely calculate the real-world probabilities. For instance, just think log-returns are normally distributed, take the mean and standard deviation of the past log-returns and ta-da... You just calculated the real world probabilities of the underlying asset! Of course, your estimate is just as good as your assumptions.

• You just assumed log-returns behaved as Normal Distribution (analogous to Brownian Motion, or very close).
• You just assumed mean and standard deviation of the asset will not change until expiration.
• You assumed several other minor things as well.

If these assumptions hold, for the time of the contract, you can calculate the real world probability of every price of the asset in the future. Your derivative's price will be equivalent to the contract function. For instance for the European call $max(S_T - K,0)$. You know the parameters and the distribution of $S_T$ and $K$ is fixed. You can then get a fair price for the contract.

What Black-Scholes does is plainly get rid of the risk parameter. Assume the drift $\mu = r - 0.5\sigma^2 + \lambda$. Suppose $\lambda$ is the risk premium/parameter. As understandable, people have various opinions about the behavior of $\lambda$. BS says that if I take a position in the underlying market with a certain size and opposite direction, I can remove $\lambda$ under certain assumptions. This is called delta hedging ($\Delta$). Since it removes all subjective parameters, you can form a replicating portfolio (i.e. it requires no money, brings no money either). If you can show you make money from trades of a replicating portfolio, it is called arbitrage. If you lose money, it is not a replicating portfolio. You can use risk neutral probabilities in real world as long as you delta hedge if it is a BS world.

The problem with BS, as with all other models, is the set of assumptions. You can say today, using a low-tails distribution to estimate log-return probabilities is ridiculous. True, it is widely used. But it is mainly for convenience, not for actual pricing (try at your peril).

Also remember Black-Scholes model came before all the theory around it. They loved the simplicity and versatility of the model so much that they thought it deserved a theory. You can read the story from here. Any serious divergence from BS model results in horribly complex and long equations which are quite confusing for the average finance people (a degree further and too complex for most people in the planet). Two years ago at a Q-Finance academic conference I witnessed a professor complimenting the presenter with the words "I did not understand what you are doing but it is beautiful" with amusement. In his defense, he was the head organizer and presenter is a distinguished honorary guest from a very reputable university.