Validating a Credit Scoring Model without Data

Question

Fellow Quants,

Suppose you have a credit scoring model that is developed without the aid of statistics, because (unfortunately) there is no historical default/loss data in your portfolio. The independent variables and weights of those variables are selected based entirely on expert judgment, and the final scores are determined as the weighted sum of the variables for each loan. i.e.

$Score_j =\sum_{i=1}^n w_i x_{ij} $

This is a very simple model, and seemingly a quite popular framework to use in the absence of loss/default data required to perform statistically-driven model development.

OSFI outlines some general principles for validating a risk rating system here, however, many of the tests require sufficient loss/default to assess the model.

How would you approach model validation for an expert judgment model in the absence of default/loss history? What type of testing can be performed when there are no "high credit risk"(e.g. defaults or losses) observations in your dataset?

Thank you,

Related question on developing a credit scoring model: Expert System for Credit Scoring

Related question on model validation criteria: Model Validation Criteria

---

EDIT: The scope of validation I have been able to come up with is (for the most part) qualitative. This would include a methodology review, assessment of the assumptions and limitations, and benchmarking (e.g. comparison of final variables against industry standard variables).

Is there an obvious component of validation that I am omitting?

Answer 1

If you don't have a significant amount of losses in your portfolio to validate the model, you should be able to obtain external loss data and adjust it where necessary to better fit your organization. This is very common with operational loss models where operational losses are quite scarce.

Answer 2

I do not know the regulatory rules for this case, but methodologically you could take another similar dataset "peer data" and then check how correctly your model predicts the losses of this dataset.

Answer 3

If you do have some positive examples to estimate your model from, then, technically, you are dealing with the task of one-class classification (a.k.a anomaly detection, also directly related to density estimation). In your case the "anomalies" are high-risk customers, not present in the data.

Various methods exist for anomaly detection and density estimation, including those based on linear models. One example would be a linear one-class SVM. Another straightforward approach would be something like a Naive-Bayes-like density estimator.

Note that all "one-class" approaches assume that your dataset is reasonably representative of the positive examples and essentially aim to discover a certain boundary around them, considering anything outside this boundary as an outlier. Consequently, blind application of this approach in your case might lead to a situation where the model would regard both "extra safe" as well as "risky" customers as "outliers". However, if you take sufficient precautions (generate additional examples which would cover the space of potentially "safe" customers, study model parameters as you do now, etc), you might get a useful model out of it.

Also, note that if you could come up with some (real-valued) "measure of riskiness" for your current data (perhaps some of the customers in your dataset are more late in their payments than others? perhaps you have external bureau credit scores for them?), you could estimate a regression model for predicting this measure and rely on it for identifying higher risk customers as well.