# Is there any other way to measure option pricing model performance than proximity to market prices?

Short version

Why do we take market prices as the prices to be estimated and predicted? The common answer is efficient markets hypothesis as in "Market agents do their best effort given their information set, therefore market prices are optimal." Is there another way?

Edit: I should add that it is the academic way of doing this. If you are to publish a paper you show how well your model represents the market compared to other models. See an example

http://www.researchgate.net/publication/222404856_GARCH_vs._stochastic_volatility_Option_pricing_and_risk_management

Long Version

Suppose I have a nice option pricing model (say Model A) to estimate some option contracts' fair prices. I use this model to estimate some of the contracts existing in the market. Let's denote the set of the price estimates as "Estimate Set A".

And let's say there is another option pricing model (say Model A) doing the same thing and get some estimates as "Estimate Set B".

And then we have the market prices since those are exchange traded options. And let's call them, well, "Market Prices".

I would like to know whether model A or model B is a 'better' option pricing model.

From what I have seen on numerous academic studies, the convention is to use an error function like root mean squared error (RMSE) and sometimes relative pricing error or some other derivation and take the Market Prices set to measure the error from. To illustrate let's say there are 4 contracts Estimate set A consists of (1, 2, 3, 4) and estimate set B consist of (4, 1, 3, 2) and market prices are (2, 3, 2, 3).

RMSE of A:

$$\sqrt{(1-2)^2+(2-3)^2+(3-2)^2+(4-3)^2} = 2$$

RMSE of B:

$$\sqrt{(4-2)^2+(1-3)^2+(3-2)^2+(2-3)^2} = \sqrt{10} \sim 3.16$$

Conclusion: A is better than B (of course it is slightly more complicated)

The only rationale I can find from the literature behind this logic is the assumption that comes from efficient markets hypothesis.

All that is required by the EMH is that investors' reactions be random and follow a normal distribution pattern so that the net effect on market prices cannot be reliably exploited to make an abnormal profit, especially when considering transaction costs (including commissions and spreads). Thus, any one person can be wrong about the market—indeed, everyone can be—but the market as a whole is always right.

Option pricing performance convention is built right atop of this hypothesis. The problem is the implicit assumption of the market price optimality. If the market prices are optimal then there is no way a model can be used as a trading strategy.

Suppose your model estimate the price of the contract as 1.5 (say dollars) and the market price of the contract is 1.2. If you gauged your model with the market prices you should accept you are off 0.3\$. So why bother with a model, even more why bother with trading?

Is there any other way?

• Your reflection of the common answer of "Why do we take market prices as the prices to be estimated and predicted?" is incorrect: We care about market prices because that is what we trade against. If you believe in emh then you should not ever engage in risk taking. But if you think market prices are NOT correctly valued then you still care about market prices because that is how the profit and loss is calculated against. – Matthias Wolf May 27 '14 at 14:43
• My arguement is neither about EMH nor from the trading perspective. The common academic approach to test a model is to measure its proximity to market and their main reason to do so is EMH. This is the criticism part, but I am looking for a solution also. – berkorbay May 27 '14 at 18:08
• I do not share the same summary of the "academic approach". I do not think that academicians consider market prices as benchmark because they believe in the EMH theory. I can again only repeat that most everyone considers market prices because they function as yardstick against which everything is measures. Even my own models, if market prices do not converge to my model then my model was obviously flawed and not the market. You look to either (a) derive a model as close to market prices as possible in order to price similar assets that may not be traded using inputs that were calibrated to.. – Matthias Wolf May 28 '14 at 7:01
• ..the market, or (b) you derive a model that makes its own asset price level/return prediction in the hope that market prices are dislocated short-term and converge to levels a model predicts. – Matthias Wolf May 28 '14 at 7:05
• I honestly don't know what is the real rationale behind it, but it is the only one I found so far. Actually it might occur from the dichotomy of the markets (options and underlying). Options market is actually and estimate on the future of the underlying's market but has its own exchange, therefore prices might diverge from the underlying's. The problem is people usually calibrate their models using underlying market and measure their model's performance using the options market. But acceptance of such arrangement is that you make an estimate over an estimate which I believe increases error. – berkorbay May 28 '14 at 10:49

This may well off-topic, but I'd like to clarify on the concept of risk-neutral pricing, which is the framework where most of the option pricing models operate. This should maybe add a perspective over the topic and this should give a partial answer to your very open-ended question.

When we are talking about option pricing, we are generally talking about "risk-neutral" pricing.

The goal is to price a product in a risk-neutral framework is to take some liquid products and put them together to replicate the payoff of some other product (look up the derivation of the B-S formula). In the business, once your pay-off has been replicated, you can hedge risk exposures through liquid products. This is particularly cornerstone in the case of a dealer of OTC products.

Perfect replication of a pay-off is not always possible. Therefore, you may want to extract "information" from liquid products to price your OTC. This is the context where I have seen the use of RMSE as a mean to gauge the quality of the calibration of your model - first you fit the model to your liquid products, then you use the model to price your OTC.

One other goal of model calibration might be extract information from a term structure, an array of securities. In this case, I have in mind, for example, local and stochastic vol models. The benefit of such models is to manage an entire portfolio of options in the present of a volatility smile/smirk.

In conclusion, risk-neutral pricing does not lead to any conclusion over market expectations. The goal is to price something and hedge away the risk with the liquid assets, available on the market right now.

• All you wrote above is right but it misses the point (good catch on the OTC but there are less painful ways to do that if we are assuming market is optimal, but let's restrict ourselves with exchange traded markets). I think I should clarify that it is an academic way of model validation. They simply say "My model represents the market option prices better (than the benchmark models) so it is a better option pricing model." I say "if you assume market is optimal, then your model will never be useful since there will always be a market price when you trade on the market." – berkorbay May 27 '14 at 19:04

You could go for a backtesting strategy and measure the p&l that trading according to your model would generate over different time intervals. Your model might be off the market prices today, but if your model is better, there will be a time-frame where it aligns again. This would be an opportunity to make money. One could say you would have found a model arbitrage opportunity.

• Although my objective is more about model validation, that is what I do now. The only difference is I try take the market error (option price vs. realized payoffs) into account for model validation and choose the best model for a specific contract to make the short-long decision, and then calculate the p&l. – berkorbay May 28 '14 at 10:54

Speaking from a practitioner's point of view, some of the these would be:

1. Model sensitivity: what happens when the price shifts by a small amount..are we still able to price correctly or do we have to adjust the model?
2. Greeks: Is the model able to calculate the greeks accurately/quickly?
3. Model "speed": How long does it take to calculate the price & greeks..one of the most important factors for practioners usually. the model should also be able to price non- vanilla instruments quickly (e.g. asian options, barrier options etc.)
4. Calibration: how many instruments are necessary to ensure a stable model environment? can we use this model for non-liquid markets?

If you are really only interested in the academic way, then I would assume that some of these points would still be quite important. Your model should still be able to calculate greeks, converge quickly and maybe most importantly, not be overparameterized.

• What is the correct price for you? In terms of options, is it the market option price or some p&l function? Or in other words how do you determine the correct (I assume fair) price? – berkorbay May 31 '14 at 13:34

I don't agree with the contention that market prices are always used as the benchmark upon which to base model performance. I think this is model dependent. Market prices make sense (for example for modelling an underlying predictor), but for example, for a derivatives model, I would argue the values of those derivatives at expiry (or earlier for path dependent derivatives) are the benchmark against which a model's performance is, or should be, compared.

I understand the paper you cited attempts to match market prices, but a reason for doing that might be to better understand why an existing model fails to match market prices, and thus better understand why a different model might be better suited, or better explain the behaviour of the market, or on the assumption that market prices are closer to observed values.

I see no particular reason why efficient market hypothesis (EMH) or random walk hypothesis (RWH) must hold. It is often convenient to assume they do when formulating models of a mathematical nature. There have been many attempts, and continue to be, to disprove RWH/EMH, and I believe the view that these do not hold (strongly) is widely adopted. Black Scholes, for example, is widely recognised as having flaws, but still invaluable - for its insight into pricing, its simple formulae, and its wide use for "implied volatility" - which is after all just the volatility implied by Black Scholes such that Black Scholes would give a particular option price.

Ways to test models' performance are in my experience typically empirical or statistical, some examples include:
- accuracy tests: how closely do model values match observed values?
- empirical tests: how well would this model perform over a given data set?
- likelihood ratio test how likely is this model to be better?
- goodness of fit tests of a statistical nature, with attempts to find distributions of results

I wonder if you found any answer to this question; would be good to know. From experience, I understood that strong form of EMH really does not apply in case of derivative products because it is very difficult to execute arbitrage strategies and hold onto those while the trades go against you in the limits and regulatory framework that most banks have. Moreover, most instruments that we calibrate the model with have large risk premiums included but still we are forced to calibrate to those instruments in order to match the market prices precisely so that we are not off market even if we believe that the market is off. Btw almost all modern models have enough free parameters to enable a broad calibration to market prices so matching the market is not really an issue.

The only approach that I could think of in such cases is to gauge the PL explanatory power of the model in terms of the risks it shows. If there is a large unexplained PL then there could be significant missing risks in the model. That I believe spawns the need to model jumps and other such factors when the BS model quite frankly works well in most cases and could be calibrated daily (a contradictory activity given that we believe yesterday's calibration was correct) to the market prices.

All you wrote are valid to the point where no real money of your own involved. Writing an academic paper is as easy as having a cup of tea. The real test of a model is actually making real bets, i.e. buying and selling options, if you make money, then the model can be said a good one!

As simple as that! Why do you have to sort to EMH or the like? Why people prefer a fantasy (EMH or risk-neutral) to a real thing, because in a fantasy everything is possible!