7

The most important questions in my opinion (besides all the mathematical questions and robustness tests that have to be performed anyway) are: Why is this model working in the first place? What is the economic rationale behind it? Why have the returns not been arbitraged away? Who is paying for your returns and why? Is this situation likely to be ...


5

I agree with many of the other comments and answers. In addition, I would recommend the paper by Gelman and Shalizi on "Philosophy and the practice of Bayesian Statistics", even if you're not a Bayesian. I would emphasize Section 4 on Model checking. They note the importance of posterior predictive checks, i.e. if I forecast from a model, would I get ...


4

Just adding the time/frequency dimension difference to what was said above: model backtesting is a model performance technique which takes place on an ongoing basis (in particular for VaR, breaches need to be discussed as soon as feasible after they take place; if there are too many over some period of time, they need to be escalated). Model validation takes ...


4

Not knowing what type of model you have, I am going to guess that these general steps will make the user of the model feel confident/comfortable about using your model. In my opinion, you have shown that your model does reasonably well: 1). If you input historical data and see if the output is reasonably close to what the real historical output was. 2). ...


3

If the model you're talking about is something that prices and risk manages an exotic (since you mentioned you calibrated to vanillas), I'd like to see: How does the evolution of the volatility surface / correlations look like. When I vega hedge, my ability to recover future prices of vanillas is important for me to not leak PnL by rebalancing/recalibrating....


2

What constitutes valid is often - a part from model risk discussed above - an agreement between subject matter experts in a specific field or company. And hence, different fields require different standards. I used to work in social science, and top journals would find things perfectly valid if Crombach Alpha, Power, R-Square, F-score and factor ...


2

"Validation" means that someone analyses the model and pronounces it fit to be used, usually subject to conditions such as ongoing performance monitoring, and restrictions on input. Good industry practices include 1 for the validator to be independent from the model developer 2 to review and re-validate important models periodically. "Ongoing ...


2

In my opinion model validation is broader than model backtesting. During model backtesting you test model performance on data that has been realised using only the data you could have used when using the model for risk management. This gives you an idea on model accuracy and it allows you to find out where the model or submodels are inaccurate or misleading. ...


1

Risk Model Validation: A practical guide to addressing the key questions by Christian Meyer and Peter Quell. The Validation of Risk Models: A Handbook for Practitioners by Sergio Scandizzo.


1

The Basel working paper contains quite a comprehensive summary of the types of metrics you are looking for: https://www.bis.org/publ/bcbs_wp14.pdf


Only top voted, non community-wiki answers of a minimum length are eligible