Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It only takes a minute to sign up.
Sign up to join this community
Let's say I have a brand new fancy model on some asset class (calibration porcedure included over a set of vanilla options) in which I truly believe I made a step forward comparing to existing literature (a phantasmatic situation I recognize).
Which criteria should be in order so I can claim that I am in a position to "validate" this model?
PS: We will assume that the math are OK and that I can prove that there is no bug in the model implementation
I don't think that there is a precise point in time when we can say that model is valid (well, it's a model not a law). For example, E. Derman in his article on Model risk describes the verification of model as a iterative process:
It is impossible to avoid errors during model development, especially when they are created under trading floor duress .... So, after the model is built, the developer tests it extensively. Thereafter, other developers “play” with it too. Next, traders who depend on the model for pricing and hedging use it. Finally, it’s released to salespeople. After a suitably long period during which most wrinkles are ironed out, it’s given to appropriate clients. This slow diffusion helps eliminate many risks, slowly but steadily.
He also describes 6 types possibilities which constitutes model risk here:
In addition to that he provides some tips that can be used to avoid model risk and its consequences (to name just few):
The most important questions in my opinion (besides all the mathematical questions and robustness tests that have to be performed anyway) are:
When you cannot satisfactorily answers these questions chances are that your results are too good to be true.
I agree with many of the other comments and answers. In addition, I would recommend the paper by Gelman and Shalizi on "Philosophy and the practice of Bayesian Statistics", even if you're not a Bayesian. I would emphasize Section 4 on Model checking. They note the importance of posterior predictive checks, i.e. if I forecast from a model, would I get sensible results. They also recommend simulating fake data assuming the model is true and comparing it to the actual data. I regularly use these tools.
Not knowing what type of model you have, I am going to guess that these general steps will make the user of the model feel confident/comfortable about using your model. In my opinion, you have shown that your model does reasonably well: 1). If you input historical data and see if the output is reasonably close to what the real historical output was. 2). Not fall into the trap of over-fitting. 3). Not use obscure reasons for calibration i.e there should be at least some mathematical justification. 4). The simple test cases are the most overlooked, make sure they are passed successfully. 5). Try to break it and explain why it did.
If the model you're talking about is something that prices and risk manages an exotic (since you mentioned you calibrated to vanillas), I'd like to see:
How does the evolution of the volatility surface / correlations look like. When I vega hedge, my ability to recover future prices of vanillas is important for me to not leak PnL by rebalancing/recalibrating.
How often does your model need to be 'recalibrated' to hit the market price over the life of the exotic? This is again problematic due to PnL leakage.
Since admittedly your model can price and risk manage things better, can you explain what features make it better than the second best model for the instrument?
-Parameter markings - If your model needs to mark parameters that are difficult to risk manage, lack intuition, or simply cannot be marked independently of others (i.e. say when one feature of the market changes, you need to change the whole family of markings to accommodate it, so you don't really know what parameter is doing which job.
Moreover, are these parameters market observable? Or can I estimate them historically?
What constitutes valid is often - a part from model risk discussed above - an agreement between subject matter experts in a specific field or company. And hence, different fields require different standards.
I used to work in social science, and top journals would find things perfectly valid if Crombach Alpha, Power, R-Square, F-score and factor significance on the regressions were within reasonable ranges.
Within risk management the bar is much higher, there is even a journal on risk model validation. And if you follow the "Cutting Edge" section in Risk you will find that academics even within one field may occasionally disagree on what is considered valid - re FVA.
To cite an other example, spatial econometrics faces different criteria in model validation.
