# Connections between logistic regression, information value and Kullback-Leibler

I am new to credit risk modeling, my background is in mathematics and physics, so I am looking to justify some commonly used techniques from first principles.

Suppose that we are interested in modeling a binary predictor $$Y=0,1$$ subject to $$m$$ predictors $$x_1,...,x_m$$. First, let us examine a simpler model of the impact of $$x_j$$ on the response $$Y$$. By the Bayes formula we have $$P(Y=1|x_j) = P(x_j|Y=1)P(Y=1)$$ $$P(Y=0|x_j) = P(x_j|Y=0)P(Y=0).$$ From these two expressions we obtain

$$\log\left(\frac{P(Y=1|x_j)}{P(Y=0|x_j)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right).$$ If we assume now that all predictors $$x_1,...,x_m$$ are independent, then we find a similar equation: $$\log\left(\frac{P(Y=1|x_1,...x_m)}{P(Y=0|x_1,...,x_m)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \sum^m_{j=1}\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right).$$

Since this is such a strong assumption to place on our predictors, we assume the Naïve Bayes form: $$\log\left(\frac{P(Y=1|x_1,...x_m)}{P(Y=0|x_1,...,x_m)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \sum^m_{j=1}\beta_j\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)$$ where the $$\beta_j$$ are estimated. The term $$\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)$$ is called the weight of evidence ($$WOE$$), and its predictive power is measured by the so-called information value defined by the following

$$IV_j = \int \log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)\left(P(x_j|Y=1)- P(x_j|Y=0) \right)dx.$$

This method is a standard technique in credit risk modeling. Many sources all over the internet (Siddiqi 2006) state the criterion
$$\text{if}\ IV< 0.02, \ \text{then useless for prediction }$$ $$\text{if}\ 0.02 $$\text{if}\ 0.1 $$\text{if}\ 0.3 $$\text{if}\ IV> 0.5, \ \text{then suspicious predictive power.}$$

Further, the information value can be written in terms of the Kullback-Leibler divergence $$IV_j = D_{KL}\left(P(x_j|Y=1)||P(x_j|Y=0)\right) + D_{KL}\left(P(x_j|Y=0)||P(x_j|Y=1)\right).$$ This being said, I have the following questions:

1. What is the Bayesian interpretation of the $$\beta_j$$? How is $$\beta_j$$ related to the dependence on the remaining variables? It is not obvious to me how one can obtain the Naïve Bayes log-formula starting from the first principles, i.e., the Bayes formula. Seems like one can't, so is there any connection to Bayesian hierarchies, correlations, ect?
2. What is the mathematical basis for the bounds on the information value and its predictive power? It is not clear why these bounds arise and thus why should they be followed. Seems to me that all references to the bounds on the information value were determined empirically.
3. Where does the information value come from? KL-divergence, $$D(P||Q)$$, can be interpreted in a number of ways, namely, as the a "distance" between two distributions. The divergence vanishes iff $$P=Q$$, so does the above criterion come from the desiring the divergence to be sufficiently small but not too small?