# Validating a Credit Scoring Model without Data

Fellow Quants,

Suppose you have a credit scoring model that is developed without the aid of statistics, because (unfortunately) there is no historical default/loss data in your portfolio. The independent variables and weights of those variables are selected based entirely on expert judgment, and the final scores are determined as the weighted sum of the variables for each loan. i.e.

$Score_j =\sum_{i=1}^n w_i x_{ij}$

This is a very simple model, and seemingly a quite popular framework to use in the absence of loss/default data required to perform statistically-driven model development.

OSFI outlines some general principles for validating a risk rating system here, however, many of the tests require sufficient loss/default to assess the model.

How would you approach model validation for an expert judgment model in the absence of default/loss history? What type of testing can be performed when there are no "high credit risk"(e.g. defaults or losses) observations in your dataset?

Thank you,

Related question on developing a credit scoring model: Expert System for Credit Scoring

Related question on model validation criteria: Model Validation Criteria

EDIT: The scope of validation I have been able to come up with is (for the most part) qualitative. This would include a methodology review, assessment of the assumptions and limitations, and benchmarking (e.g. comparison of final variables against industry standard variables).

Is there an obvious component of validation that I am omitting?

If you don't have a significant amount of losses in your portfolio to validate the model, you should be able to obtain external loss data and adjust it where necessary to better fit your organization. This is very common with operational loss models where operational losses are quite scarce.

• I'm familiar with this approach, but I've only ever used it when 'some' default/loss data was available. Otherwise we'd essentially have an external distribution that we would 'adjust' to 'better fit' a distribution that we don't know. Perhaps you have a resource? I appreciate your response, thank you. – dmanuge Nov 26 '15 at 23:13
• Sorry, I don't have a resource, try searching on here for a similar question. Failing that, ask it and hopefully someone does. I'm a bit out of my depth here, but I don't think you would adjust the external distribution to fit your internal one. What you might do is remove data points that are not relevant example losses from structured products if you don't have those in your portfolio. – AfterWorkGuinness Nov 26 '15 at 23:29
• So you're suggesting to make adjustments based on qualitative characteristics of your portfolio, rather than quantitative. I agree this can be done to some extent, and works better when you have complete information on your external distribution, but lending practices and idiosyncratic risks can differ a lot between firms. You would have to find external data for comparable firms, which make the problem more difficult (e.g. competitors tend not to share their internal data). – dmanuge Nov 27 '15 at 16:27
• As for fitting internal distributions to external data, you may be interested in this presentation: bostonfed.org/2015stm/gordon_liu.pdf – dmanuge Nov 27 '15 at 16:27
• Hi dmanuge, I'm not suggesting firms openly publish their loss distributions (I don't think they do, not certain, but doubt it). I do believe there are sources you can find/buy this data from. It would be a loss distribution built using data from multiple similar firms e.g.: US mid size manufacturing. Sorry I can't be more specific, but hopefully this points you in the right direction. – AfterWorkGuinness Dec 7 '15 at 18:18

I do not know the regulatory rules for this case, but methodologically you could take another similar dataset "peer data" and then check how correctly your model predicts the losses of this dataset.

If you do have some positive examples to estimate your model from, then, technically, you are dealing with the task of one-class classification (a.k.a anomaly detection, also directly related to density estimation). In your case the "anomalies" are high-risk customers, not present in the data.

Various methods exist for anomaly detection and density estimation, including those based on linear models. One example would be a linear one-class SVM. Another straightforward approach would be something like a Naive-Bayes-like density estimator.

Note that all "one-class" approaches assume that your dataset is reasonably representative of the positive examples and essentially aim to discover a certain boundary around them, considering anything outside this boundary as an outlier. Consequently, blind application of this approach in your case might lead to a situation where the model would regard both "extra safe" as well as "risky" customers as "outliers". However, if you take sufficient precautions (generate additional examples which would cover the space of potentially "safe" customers, study model parameters as you do now, etc), you might get a useful model out of it.

Also, note that if you could come up with some (real-valued) "measure of riskiness" for your current data (perhaps some of the customers in your dataset are more late in their payments than others? perhaps you have external bureau credit scores for them?), you could estimate a regression model for predicting this measure and rely on it for identifying higher risk customers as well.