Connections between logistic regression, information value and Kullback-Leibler

Question

I am new to credit risk modeling, my background is in mathematics and physics, so I am looking to justify some commonly used techniques from first principles.

Suppose that we are interested in modeling a binary predictor $Y=0,1$ subject to $m$ predictors $x_1,...,x_m$. First, let us examine a simpler model of the impact of $x_j$ on the response $Y$. By the Bayes formula we have $$P(Y=1|x_j) = P(x_j|Y=1)P(Y=1) $$ $$P(Y=0|x_j) = P(x_j|Y=0)P(Y=0). $$ From these two expressions we obtain

$$\log\left(\frac{P(Y=1|x_j)}{P(Y=0|x_j)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right). $$ If we assume now that all predictors $x_1,...,x_m$ are independent, then we find a similar equation: $$\log\left(\frac{P(Y=1|x_1,...x_m)}{P(Y=0|x_1,...,x_m)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \sum^m_{j=1}\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right). $$

Since this is such a strong assumption to place on our predictors, we assume the Naïve Bayes form: $$\log\left(\frac{P(Y=1|x_1,...x_m)}{P(Y=0|x_1,...,x_m)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \sum^m_{j=1}\beta_j\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)$$ where the $\beta_j$ are estimated. The term $\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)$ is called the weight of evidence ($WOE$), and its predictive power is measured by the so-called information value defined by the following

$$IV_j = \int \log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)\left(P(x_j|Y=1)- P(x_j|Y=0) \right)dx. $$

This method is a standard technique in credit risk modeling. Many sources all over the internet (Siddiqi 2006) state the criterion
$$\text{if}\ IV< 0.02, \ \text{then useless for prediction } $$ $$\text{if}\ 0.02<IV< 0.1, \ \text{then weak predictive power } $$ $$\text{if}\ 0.1<IV< 0.3, \ \text{then medium predictive power } $$ $$\text{if}\ 0.3<IV< 0.5, \ \text{then strong predictive power } $$ $$\text{if}\ IV> 0.5, \ \text{then suspicious predictive power.} $$

Further, the information value can be written in terms of the Kullback-Leibler divergence $$IV_j = D_{KL}\left(P(x_j|Y=1)||P(x_j|Y=0)\right) + D_{KL}\left(P(x_j|Y=0)||P(x_j|Y=1)\right).$$ This being said, I have the following questions:

1. What is the Bayesian interpretation of the $\beta_j$? How is $\beta_j$ related to the dependence on the remaining variables? It is not obvious to me how one can obtain the Naïve Bayes log-formula starting from the first principles, i.e., the Bayes formula. Seems like one can't, so is there any connection to Bayesian hierarchies, correlations, ect?
2. What is the mathematical basis for the bounds on the information value and its predictive power? It is not clear why these bounds arise and thus why should they be followed. Seems to me that all references to the bounds on the information value were determined empirically.
3. Where does the information value come from? KL-divergence, $D(P||Q)$, can be interpreted in a number of ways, namely, as the a "distance" between two distributions. The divergence vanishes iff $P=Q$, so does the above criterion come from the desiring the divergence to be sufficiently small but not too small?

Thank you for your time.

Answer 1

With apologies for lowering the tone of your well framed question, but my casual interpretation from a major credit risk project (2003-2005) indirectly advised by Siddiqi, was that the IV > 0.5 "suspiciously high" bracket was a crude sanity check flag that the predictor had an unnatural relationship with the response possibly caused by (basically) mistakes in the way the analysts had assembled the data.

I could give a couple of examples of such although that would digress.

We were calculating IV on each predictor individually as a screening stage. Predictors were categorical, having been grouped or bucketed where necessary. The process followed Siddiqi's notes (later, book):

Siddiqi, Naeem. 2005. Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring. Hoboken: Wiley.