Risk Model Validation

Question

I have such a general question regarding risk model validation. Which tools are most often used for validation and how does the process work? Could you recommend any books that focus on this topic?

Answer 1

You should read this regulatory guidance:

U.S.:

SR 11-7:

https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf

(it is identical to FHFA AB 2013-07 Model Risk Management Guidance, OCC Bulletin 2011-12, https://www.fdic.gov/news/financial-institution-letters/2017/fil17022a.pdf , et al)

OCC brochure on model risk management (August 2021)

Canada (very similar):

OSFI Enterprise-Wide Model Risk Management for Deposit-Taking Institutions E23 https://www.osfi-bsif.gc.ca/Eng/Docs/E23.pdf

U.K.:

PRA. Supervisory Statement SS3/18. Model risk management principles for stress testing (April 2018) https://www.bankofengland.co.uk/-/media/boe/files/prudential-regulation/supervisory-statement/2018/ss318.pdf (although the focus is on stress testing, much of it applies to models in general.)

Bank of England. Consultation Paper 6/22 – Model risk management principles for banks. https://www.bankofengland.co.uk/prudential-regulation/publication/2022/june/model-risk-management-principles-for-banks

and these 3 books:

Massimo Morini. Understanding and Managing Model Risk: A Practical Guide for Quants, Traders and Validators. Wiley (2011)

Christian Meyer, Peter Quell. Risk Model Validation. Risk Books (3rd edition, 2020)

Sergio Scandizzo. The Validation of Risk Models: A Handbook for Practitioners. Palgrave Macmillan (2016)

Other related resources worth mentioning:

American Academy of Actuaries. Model Risk Management: A Public Policy Practice Note (May 2019).

Model Risk Managers' International Association. White papers. https://mrmia.org/white-papers/ See also MRMIA Best Practices: Volume 1

Journal of Risk Model Validation https://www.risk.net/journal-of-risk-model-validation (subscription needed)

Los Alamos National Laboratory. Ben Thacker et al. Concepts of Model Verification and Validation (V&V). 2004. LA-14167-MS https://doi.org/10.2172/835920 (This older paper greatly influenced SR 11-7 and other regulatory guidance.)

Seppe vanden Broucke and Bart Baesens. Managing Model Risk: Lessons and experiences from industry and research on the challenges and dangers of analytical models (2021)

Aruna Joshi (Visa). Managing Risk of Financial Models: A Smart and Simple Guide for the Practitioner (2017)

George Christodoulakis and Stephen Satchell, Editors. The Analytics of Risk Model Validation (Quantitative Finance) (2007) (This is a collection of narrowly focused articles, somewhat dated.)

Daniel Rösch and Harald Scheule, editors. Model Risk: Identification, Measurement and Management. Risk Books (2010) (Another collection of narrowly focused articles.)

Answer 2

Model Validation process usually consists of:

1. Conceptual Soundness Review (model assumptions, mathematical representation, limitations)

• Here you should try to re-derive the model from scratch and ask yourself what the assumptions that you are making along the way are, which should then tell you what the model limitations are. Alternatively if you don't want to rederive the model from scratch / can't / can't due to time constraints, you can go through the maths provided by the model developer and check all the assumptions with a focus on model suitability.

2. Testing (through reimplementation or developer's interface)

• Here, you test the model implementation. Ideally, your model validation team will have an independent library where you test the model independently and see whether it agrees with the developer's implementation. If the model validation team doesn't have its own library, the developer will provide an interface where you can at least try some "corner" scenarios and see if the model performs as expected (if it's a pricing model, how does it behave when volatility or rates go to zero, how does it behave when volatility or rates become very large, etc.).

3. Assessment of Model Documentation (written by model developer)

• This might be frowned upon, but a big part of model validator's job is to make sure the model documentation (provided by the developers) is spot on and up to scratch. In other words, model validators job is to help improve the front-office model developer's documentation. You can ask for more testing to be done and documented if the documented testing is not sufficient. You should also check that all graph-axes are being clearly labelled, that variable names are consistent throughout the doc, etc (these things are "trivial" but very important).

4. Writing a self-contained Validation Document describing the Model

• Finally, once all of the above is done, the model validator should produce an independent doc where all his / her work is well documented. This should mainly include an independent derivation of the model and the independent testing completed, and it should clearly state all model limitations (i.e. deterministic volatility model cannot be used to price forward-starting options, etc.).

Attention to detail and ability to challenge the FO quants constructively and politely are critical skills here: if the challenges are "stupid" or the FO quants feel that you're not trying to help, the long-term cooperation will not be very productive. On the other hand, if you ask smart questions and try to work together, the work with FO developers can be enjoyable and a model-val role can be a good opportunity to do deep-dives into specific models and increase your understanding of these models.