Let's say I have a brand new fancy model on some asset class (calibration porcedure included over a set of vanilla options) in which I truly believe I made a step forward comparing to existing literature (a phantasmatic situation I recognize).

Which criteria should be in order so I can claim that I am in a position to "validate" this model?

PS: We will assume that the math are OK and that I can prove that there is no bug in the model implementation

  • 8
    $\begingroup$ I am not a professional quant but I am a professional simulation modeler. I can tell you that a validation process is something that is agreed upon by all stakeholders, as there are almost infinitely many metrics and aspects of any non-trivial model that may be examined. In a nutshell, it's valid when you say it's valid, according to your standards. $\endgroup$
    – Pete
    Feb 4 '11 at 16:16
  • $\begingroup$ @Pete What are the standards under which you would agree that a given model is valid? $\endgroup$
    – user1157
    Feb 15 '14 at 11:50
  • $\begingroup$ If the OP would specify which type of models and which field he is interested in, perhaps more meaningful answers could be provided. I could suggest a few items that I look at for regression models for example. $\endgroup$
    – RndmSymbl
    Feb 15 '14 at 16:18
  • 1
    $\begingroup$ @Raphael looking again at the OP, he says "made a step forward comparing to existing literature". So in this case, suitable validation means his model can reproduce the results/subset of results in the existing literature. $\endgroup$
    – Pete
    Feb 18 '14 at 2:35

I don't think that there is a precise point in time when we can say that model is valid (well, it's a model not a law). For example, E. Derman in his article on Model risk describes the verification of model as a iterative process:

It is impossible to avoid errors during model development, especially when they are created under trading floor duress .... So, after the model is built, the developer tests it extensively. Thereafter, other developers “play” with it too. Next, traders who depend on the model for pricing and hedging use it. Finally, it’s released to salespeople. After a suitably long period during which most wrinkles are ironed out, it’s given to appropriate clients. This slow diffusion helps eliminate many risks, slowly but steadily.

He also describes 6 types possibilities which constitutes model risk here:

  • Incorrect model
  • Correct model, incorrect solution
  • Correct model, inappropriate use
  • Badly approximated solution
  • Software and hardware bugs
  • Unstable data

In addition to that he provides some tips that can be used to avoid model risk and its consequences (to name just few):

  • Test complex models in simple cases first
  • Test the model’s boundaries
  • Don’t ignore small discrepancies
  • 1
    $\begingroup$ We view validation and verification as two separate processes. "Verification": does the program and model do what it is supposed to/designed to do? E.g. if it is supposed to trade when condition x is true, does it? "Validation": does it predict well enough to make money. $\endgroup$
    – Pete
    Feb 5 '11 at 18:48
  • $\begingroup$ Well I know this paper and I find it really unsatisfactory. Anyway it is principally about operational risk in implementing a model (case that I discarded in my question) rather than model validation in my opinion. Anyway when you've finished reading it you still don't realy know what to do exactly with your new model to know how it can be validated. $\endgroup$
    – TheBridge
    Feb 5 '11 at 21:03

The most important questions in my opinion (besides all the mathematical questions and robustness tests that have to be performed anyway) are:

  • Why is this model working in the first place?
  • What is the economic rationale behind it?
  • Why have the returns not been arbitraged away?
  • Who is paying for your returns and why?
  • Is this situation likely to be sustainable?

When you cannot satisfactorily answers these questions chances are that your results are too good to be true.

  • 1
    $\begingroup$ I like that point of view. $\endgroup$
    – Svisstack
    Feb 20 '14 at 17:42

I agree with many of the other comments and answers. In addition, I would recommend the paper by Gelman and Shalizi on "Philosophy and the practice of Bayesian Statistics", even if you're not a Bayesian. I would emphasize Section 4 on Model checking. They note the importance of posterior predictive checks, i.e. if I forecast from a model, would I get sensible results. They also recommend simulating fake data assuming the model is true and comparing it to the actual data. I regularly use these tools.


Not knowing what type of model you have, I am going to guess that these general steps will make the user of the model feel confident/comfortable about using your model. In my opinion, you have shown that your model does reasonably well: 1). If you input historical data and see if the output is reasonably close to what the real historical output was. 2). Not fall into the trap of over-fitting. 3). Not use obscure reasons for calibration i.e there should be at least some mathematical justification. 4). The simple test cases are the most overlooked, make sure they are passed successfully. 5). Try to break it and explain why it did.

  • $\begingroup$ I especially like 4) and 5): +1 $\endgroup$
    – vonjd
    Feb 14 '14 at 9:04

If the model you're talking about is something that prices and risk manages an exotic (since you mentioned you calibrated to vanillas), I'd like to see:

  • How does the evolution of the volatility surface / correlations look like. When I vega hedge, my ability to recover future prices of vanillas is important for me to not leak PnL by rebalancing/recalibrating.

  • How often does your model need to be 'recalibrated' to hit the market price over the life of the exotic? This is again problematic due to PnL leakage.

  • Since admittedly your model can price and risk manage things better, can you explain what features make it better than the second best model for the instrument?

-Parameter markings - If your model needs to mark parameters that are difficult to risk manage, lack intuition, or simply cannot be marked independently of others (i.e. say when one feature of the market changes, you need to change the whole family of markings to accommodate it, so you don't really know what parameter is doing which job.

Moreover, are these parameters market observable? Or can I estimate them historically?

  • Implementation speed of course is important.

What constitutes valid is often - a part from model risk discussed above - an agreement between subject matter experts in a specific field or company. And hence, different fields require different standards.

I used to work in social science, and top journals would find things perfectly valid if Crombach Alpha, Power, R-Square, F-score and factor significance on the regressions were within reasonable ranges.

Within risk management the bar is much higher, there is even a journal on risk model validation. And if you follow the "Cutting Edge" section in Risk you will find that academics even within one field may occasionally disagree on what is considered valid - re FVA.

To cite an other example, spatial econometrics faces different criteria in model validation.

  • $\begingroup$ Apart from your point that all seems to be relative could you give some criteria that you think would be helpful in the context of quantitative trading (let's stick with the buy side for the sake of simplicity). $\endgroup$
    – vonjd
    Feb 16 '14 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.