Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

Let's say I have a brand new fancy model on some asset class (calibration porcedure included over a set of vanilla options) in which I truly believe I made a step forward comparing to existing literature (a phantasmatic situation I recognize).

Which criteria should be in order so I can claim that I am in a position to "validate" this model?

PS: We will assume that the math are OK and that I can prove that there is no bug in the model implementation

share|improve this question
8  
I am not a professional quant but I am a professional simulation modeler. I can tell you that a validation process is something that is agreed upon by all stakeholders, as there are almost infinitely many metrics and aspects of any non-trivial model that may be examined. In a nutshell, it's valid when you say it's valid, according to your standards. –  Pete Feb 4 '11 at 16:16
    
@Pete What are the standards under which you would agree that a given model is valid? –  user1157 Feb 15 at 11:50
    
If the OP would specify which type of models and which field he is interested in, perhaps more meaningful answers could be provided. I could suggest a few items that I look at for regression models for example. –  RndmSymbl Feb 15 at 16:18
1  
@Raphael looking again at the OP, he says "made a step forward comparing to existing literature". So in this case, suitable validation means his model can reproduce the results/subset of results in the existing literature. –  Pete Feb 18 at 2:35

5 Answers 5

I don't think that there is a precise point in time when we can say that model is valid (well, it's a model not a law). For example, E. Derman in his article on Model risk describes the verification of model as a iterative process:

It is impossible to avoid errors during model development, especially when they are created under trading floor duress .... So, after the model is built, the developer tests it extensively. Thereafter, other developers “play” with it too. Next, traders who depend on the model for pricing and hedging use it. Finally, it’s released to salespeople. After a suitably long period during which most wrinkles are ironed out, it’s given to appropriate clients. This slow diffusion helps eliminate many risks, slowly but steadily.

He also describes 6 types possibilities which constitutes model risk here:

  • Incorrect model
  • Correct model, incorrect solution
  • Correct model, inappropriate use
  • Badly approximated solution
  • Software and hardware bugs
  • Unstable data

In addition to that he provides some tips that can be used to avoid model risk and its consequences (to name just few):

  • Test complex models in simple cases first
  • Test the model’s boundaries
  • Don’t ignore small discrepancies
share|improve this answer
    
We view validation and verification as two separate processes. "Verification": does the program and model do what it is supposed to/designed to do? E.g. if it is supposed to trade when condition x is true, does it? "Validation": does it predict well enough to make money. –  Pete Feb 5 '11 at 18:48
    
Well I know this paper and I find it really unsatisfactory. Anyway it is principally about operational risk in implementing a model (case that I discarded in my question) rather than model validation in my opinion. Anyway when you've finished reading it you still don't realy know what to do exactly with your new model to know how it can be validated. –  TheBridge Feb 5 '11 at 21:03

The most important questions in my opinion (besides all the mathematical questions and robustness tests that have to be performed anyway) are:

  • Why is this model working in the first place?
  • What is the economic rationale behind it?
  • Why have the returns not been arbitraged away?
  • Who is paying for your returns and why?
  • Is this situation likely to be sustainable?

When you cannot satisfactorily answers these questions chances are that your results are too good to be true.

share|improve this answer
1  
I like that point of view. –  Svisstack Feb 20 at 17:42

Not knowing what type of model you have, I am going to guess that these general steps will make the user of the model feel confident/comfortable about using your model. In my opinion, you have shown that your model does reasonably well: 1). If you input historical data and see if the output is reasonably close to what the real historical output was. 2). Not fall into the trap of over-fitting. 3). Not use obscure reasons for calibration i.e there should be at least some mathematical justification. 4). The simple test cases are the most overlooked, make sure they are passed successfully. 5). Try to break it and explain why it did.

share|improve this answer
    
I especially like 4) and 5): +1 –  vonjd Feb 14 at 9:04

What constitutes valid is often - a part from model risk discussed above - an agreement between subject matter experts in a specific field or company. And hence, different fields require different standards.

I used to work in social science, and top journals would find things perfectly valid if Crombach Alpha, Power, R-Square, F-score and factor significance on the regressions were within reasonable ranges.

Within risk management the bar is much higher, there is even a journal on risk model validation. And if you follow the "Cutting Edge" section in Risk you will find that academics even within one field may occasionally disagree on what is considered valid - re FVA.

To cite an other example, spatial econometrics faces different criteria in model validation.

share|improve this answer
    
Apart from your point that all seems to be relative could you give some criteria that you think would be helpful in the context of quantitative trading (let's stick with the buy side for the sake of simplicity). –  vonjd Feb 16 at 10:25

I agree with many of the other comments and answers. In addition, I would recommend the paper by Gelman and Shalizi on "Philosophy and the practice of Bayesian Statistics", even if you're not a Bayesian. I would emphasize Section 4 on Model checking. They note the importance of posterior predictive checks, i.e. if I forecast from a model, would I get sensible results. They also recommend simulating fake data assuming the model is true and comparing it to the actual data. I regularly use these tools.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.