I am working on a credit scoring model for a credit without a garanty (consumer credit).

I'm looking for a methodology for an expert system for credit scoring decision that doesn't base on statistical methods. I tried several statistical methods on my data (logistic regression, random forest) - without sucess. My variables are not predictive enough and predictions are too poor to build a scorecard.

I am modeling a binary variable with levels 0 and 1 where 1 is the default and I have several predictive variables.

Now I'm looking for a method that combines the information of the data with an expert system (human knowledge)? What can I do?

Does a method/algorithm like this exists? What about Apriori?

  • $\begingroup$ Hi @Charlotte and welcome to quant.SE! The judgemental credit scoring models, as the statistical one too, depend on the kind of instrument you're going to model. Could you specify the credit instrument you've to model and the specification model you used for by editing the question, please? $\endgroup$
    – Quantopik
    Jul 10 '15 at 13:52

The best model used by practitioners to model consumer credit is the logistic regression model; now, you said that you tried modelling by using logistic model without success, but I want trying to suggest you the step-by-step procedure I would use in such case.

To model consumer credit, I'd follow the following procedure:

  • Sample Analysis:

I analyze the distribution of such sample, verifying that it has approximately the same distribution of the population or benchmark. The sample size should be adequate (at least 50k observations over at least 5/7 years observations, according to what is suggested by Basel agreement).

In such step, you can perform this kind of analysis by using the Population Stability Index, given, for each variable, by:

$PSI$ $=$ $\sum\limits_{i=1}^n (P_{i,1}-P_{i,2})*ln\frac{P_{i,1}}{P_{i,2}}$

where $P_{i,1}$ is the percentage of observations in the j-class of the population/benchmark you considered and $P_{i,2}$ is the the percentage of observations in the j-class of the sample.

Compute the PSI for each variable in your sample, by verifying that the PSI is less than 10%; the threshold is empirical and depends on the country/instrument you're analyzing.

Eliminate from the sample variables that are not stable on the basis of the PSI.

  • Data Quality Analysis:

analyze the sample values pinpointing the percentage of missing values, anomalies in the values (as, for instance, value 999.999) and error values according to the economic sense (as, for instance, negative values for variables that should be necessarily positive, like the loan value). I'd eliminate all variables with missings, anomalies or error values with a percentage greater than 5%.

  • Logistic Regression Analysis

Implement a logistic by regressing the target variable on the independent ones, by eliminating the variables that are not significative at the 5% level; eliminate the variables with correlation greater than 30% (again, this is an empirical rule, so, use that as rule of thumbs and not as threshold point). After that, run a stepwise variable selection model, by keeping the significant variables only.

Now, this is the basic step-by-step procedure I'd use to model consumer credit market. Following this one, you should get an AUROC about equal to 80% (70% on the test/validation set). If it is not, you could calibrate/adjust your model by introducing expert/judgemental variables in. This means asking your colleagues to define what variables, according to them, influences the probability of default and adjust the weight the variables have in the model by capping them; this is the simpler way. Secondly, you could develop a parallel model constructed on judgemental way entirely (for instance, make a questionnaire about the variables could affect the PD and ask to several practitioners to compile that and, finally, take the more significative variables.

If you're not a practitioner, the following ones are the most common variables affecting the PD:

  • Loan/Credit amount;
  • Debt-To-Income ratio;
  • Age;
  • Job;
  • Wage/salary amount;
  • Credit Bureau score;
  • Past consumer hystory (if the customer bought something in the past and was be able to pay back or not the debt);

and so on... Unfortunately, it does not exist an algorithm to develop expert-based credit risk model. Indeed, the expert judge is based on the experience of the bank teller/manager and such experience changes according to the country in which the employer. The consumer credit market changes according to a lot of things, such as the country culture, the rules, the law, the way the Central Banks applies Basel agreement, and so on. It is not possible to adapt a model build to perform in UK to a Latin country, like Spain, Italy, etc

Hope this helps.

  • 1
    $\begingroup$ Many thanks for this answer! Indeed I have only 5000 observations with 600 defaults and my variables are not so predictive. I have an accuracy of 25%. You wrote " adjust the weight the variables have in the model by capping them". What is capping? How can I adjust the weight? $\endgroup$
    – Charlotte
    Jul 11 '15 at 11:37
  • 1
    $\begingroup$ @Charlotte: It is pretty difficult to build a predictive model in such context; your dataset is not balanced, since you should theoretically have the half of observations as defaults (1) and the remaining half as bonis (0). So, try to reduce the observations number in order to achieve a partial balance (40% default and 60% of bonis should works well anyway). Instead, capping is statistical technique used to calibrate the model; it consists to block a $\beta$ for a variable too much significant on a particular value. Do you have SAS license on your PC? $\endgroup$
    – Quantopik
    Jul 11 '15 at 11:52
  • $\begingroup$ Please, in the case the answer fulfills your question, please mark that with the check @Charlotte! $\endgroup$
    – Quantopik
    Jul 11 '15 at 11:55
  • $\begingroup$ ok, no I don't have SAS; is it possible with R? $\endgroup$
    – Charlotte
    Jul 11 '15 at 13:44
  • $\begingroup$ I didn't do that in R @Charlotte, but I know you can do that by using the package "anesrake"; alternatively, you can do it manually, by setting $\beta_k$ equal to a value and seeing if AUROC/Accuracy ratio increases (running again the stepwise variable selection procedure each time). Anyway, balancing the sample, IMHO, remains the optimal (and simpler) solution. $\endgroup$
    – Quantopik
    Jul 11 '15 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.