11 votes
Accepted

Risk Model Validation

You should read this regulatory guidance: U.S.: SR 11-7: https://www.federalreserve.gov/supervisionreg/srletters/sr1107a1.pdf (it is identical to FHFA AB 2013-07 Model Risk Management Guidance, OCC ...
user avatar
7 votes

why does Cross Validation *not* solve Backtest overfitting?

If they publish information about all K trials, then you're right. But the author's point is that that's not typical practice. Typical practice is to not disclose that information, and it amounts to p-...
user avatar
  • 1,470
5 votes

Risk Model Validation

Model Validation process usually consists of: 1. Conceptual Soundness Review (model assumptions, mathematical representation, limitations) Here you should try to re-derive the model from scratch and ...
user avatar
  • 4,961
4 votes

Backtesting of Risk models

Just adding the time/frequency dimension difference to what was said above: model backtesting is a model performance technique which takes place on an ongoing basis (in particular for VaR, breaches ...
user avatar
  • 4,963
4 votes

Validating a Credit Scoring Model without Data

If you don't have a significant amount of losses in your portfolio to validate the model, you should be able to obtain external loss data and adjust it where necessary to better fit your organization. ...
user avatar
4 votes

Model Validation Criteria

If the model you're talking about is something that prices and risk manages an exotic (since you mentioned you calibrated to vanillas), I'd like to see: How does the evolution of the volatility ...
user avatar
  • 1,662
3 votes

Validating a Credit Scoring Model without Data

I do not know the regulatory rules for this case, but methodologically you could take another similar dataset "peer data" and then check how correctly your model predicts the losses of this dataset.
user avatar
  • 5,629
3 votes
Accepted

Model Validation Aggregation Documentation (Binomial, Hosmer-Lemeshow, Tolerance) - Credit Risk et cetera

Take a look at these: Bauke Maarse. Master Thesis: Backtesting Framework for PD, EAD and LGD (2012) https://essay.utwente.nl/61905/1/master_B._Maarse.pdf Fábio Yasuhiro Tsukahara, Herbert Kimura, ...
user avatar
2 votes

How to Validate and Test a Discount curve (i.e. SOFR, LIBOR, ESTR)

A first step would obvisouly be to check if the curve you built replicates the input instruments. A second step might be to check the forwards to see if there is irregular behaviour around the curve ...
user avatar
  • 5,285
2 votes

Validating a Credit Scoring Model without Data

If you do have some positive examples to estimate your model from, then, technically, you are dealing with the task of one-class classification (a.k.a anomaly detection, also directly related to ...
user avatar
  • 75
2 votes

Backtesting of Risk models

"Validation" means that someone analyses the model and pronounces it fit to be used, usually subject to conditions such as ongoing performance monitoring, and restrictions on input. Good ...
user avatar
2 votes

Backtesting of Risk models

In my opinion model validation is broader than model backtesting. During model backtesting you test model performance on data that has been realised using only the data you could have used when using ...
user avatar
  • 7,598
1 vote
Accepted

Book suggestions for model validation (Gini, Somers D, Kolmogorov Smirnov, Kendal's Tau, Binomial/Adjusted binomial test etc)

Risk Model Validation: A practical guide to addressing the key questions by Christian Meyer and Peter Quell. The Validation of Risk Models: A Handbook for Practitioners by Sergio Scandizzo.
user avatar
1 vote

Book suggestions for model validation (Gini, Somers D, Kolmogorov Smirnov, Kendal's Tau, Binomial/Adjusted binomial test etc)

The Basel working paper contains quite a comprehensive summary of the types of metrics you are looking for: https://www.bis.org/publ/bcbs_wp14.pdf
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible