# Problem of PD estimation [closed]

Why is small number of defaults is a problem in case of PD estimation? What are the consequences? Can you recommend notes, books, etc about the topic?

• The fewer defaults you have the more imprecise your estimate of the PD. The worst case is when there have been no defaults: a naive estimate is $PD=0$ but common sense tells us that cannot be true: sooner or later there will be some defaults if we follow enough companies for a long enough time. "It has not happened yet" is not the same es "it will never happen". – Alex C Jul 2 '17 at 17:10

A low default portfolio (LDP) problem is encountered more often in case of wholesale lending portfolios where the number of borrowers is much less compared to retail lending portfolios where the number of borrowers run into tens of thousands and consequently low default portfolios are often the exception. A useful presentation on negotiating low default portfolios can be found here. This presentation has references to other important work done on working with LDPs.

So there are two issues present in estimating the probability of default, the scarcity of observed defaults and the form of the model. For simplicity, we will restrict ourselves to Pearson-Neyman decision theory and Bayesian decision theory. Presumably, if you are calculating a default rate or a probability of default, you are going to do something with this information. If that is not the case, as may be true in academic research, then you should note that the likelihood-based solution will be similar to the Frequentist solution anyway.

@AlexC points out the obvious extreme case, that the number of observed defaults is zero. If you used a simple counting mechanism such as defaults/total observations then in a Frequentist methodology, the projected probability of default is zero. It won't happen. Don't worry about it. Of course, this is naive, as is also pointed out by @AlexC. The Bayesian alternative is problematic in that a solution is not unique. The solution will depend entirely upon the prior alone. Although the data will influence the final solution, the posterior estimate will be heavily determined by the estimate prior to collecting any data.

To put this in a real-world context, consider a firm that had 1,000 credit transactions with customers and no defaults. For simplicity, assume that if there is a default then there is no possibility of partial recovery. That simplicity has to go away, but let's keep this simple. The firm is seeking a bank loan to expand and the bank will do so but rejects their accounting department's "allowance for doubtful debt." They want compensating balances to cover the allowance.

The problem of using a Bayesian method is that you have to state your beliefs about the default rate, prior to collecting data. Of course, this could be retrospective and one could say "last year I believed the default rate would be $\mu\pm\sigma$.

The question is what density function to use. Someone could use the uniform distribution as the prior distribution, which is the same thing as saying all possible default rates are equally possible. The difficulty with this solution is that it probably is not credible. If something is a rare event, then probably everyone knows it is a rare event. In fact, if you believed that there was an equal chance of 999 defaults and 1 default, then your pricing would probably drive all customers out of the credit market.

So let's consider a simpler prior density, where any default rate $\theta$ is more probable to the left and less probable to the right. The easiest version of this is the triangle distribution. It's density is $2(1-\theta).$ This is tempting because all it says is small values are more probable than large values. The challenge is that the prior expected rate of default is 33%, which is not a rare event.

Alternatively, one could ask the managers what they expected. Let's assume the manager said that from prior experience they expected a one percent, plus or minus one percent rate of default. Using the beta distribution as a model, that is approximately a beta distribution of one default per ninety-seven successes. The prior density is $97(1-\theta)^{96}$.

If you have not noticed, we haven't even begun to discuss things like logistic regression.

So now let's consider our four possible default rate calculations, the Frequentist, the uniform distribution, the triangular distribution and the density from an expert.

Under the Frequentist solution, the probability of default is zero and the confidence interval is zero in width so it is [0,0].

The Bayesian estimate using the uniform is one default per 1002 transactions. The Bayesian 95% highest density credible interval is (0.00253%,.368%).

The Bayesian estimate using the triangular distribution is one default per one thousand and three observations. The Bayesian 95% highest density credible interval is (0.00253%,.367%)

The Bayesian estimate using the expert managerial opinion is one per one thousand and ninety-eight observations. The Bayesian 95% highest density credible interval is (0.00231%, .336%).

Of course, the deeper issue is that we have estimated a parameter on an event we have never even seen once.

Let's assume they got the bank loan, and there were 2,000 total observations the next year and nine were defaults. Strangely, the results must depart because the Frequentist solution is to treat this as a new sample. While you could combine them in a meta-analysis, or if you were doing an analysis of the entire set, the default is to separate them as separate samples. Repetition is important in Frequentist theory. We will solve it both ways, however, since someone approaching the problem for the first time at the end of the second year wouldn't see the first year as a different sample.

The second year estimate would be .45% with a confidence interval of (.430%,.470%).

The joint year estimate would be .3% with a confidence interval of (.289%,.311%).

To simplify, only the "expert" decision will be tested. The mean parameter estimate is .323%. The 95% highest density credible interval is (.155%,.551%). The last Bayesian posterior would become the prior for the new estimate.

You might ask yourself, "why is the Bayesian estimate so wide compared to the Frequentist?" The Bayesian method basically takes the uncertainty around rare events as more substantial than Frequentist methods would imply. While the Frequentist joint estimate is quite tight with a width of .022%, the Bayesian is quite wide with a width of .396%. That is 18 times wider than the Frequentist method.

The difference is created by how the two methods interpret uncertainty. The Frequentist assumes the true model is known and since it is known only chance remains. The Bayesian method does not assume the true model is known and the infinite set of possible true models is assigned a probability distribution that it is the true model.

Basically, a Frequentist prediction is a prediction built on an expectation, while the Bayesian method provides an expectation of a prediction. P(E(X)) versus E(P(X)).

If the probability had been nearer to 50% then the Bayesian interval would tend to be inside the Frequentist interval because you have seen so many examples of both cases.

Which of the two should you consider using? If your question is of a true or false nature, such as $H_0:.1\le\theta$, then you should use the Frequentist solution. On the other hand, if the true value of $\theta$ matters, particularly if you are gambling on the result as you would as the management creating a policy for recievables, then you should use the Bayesian method.

The reason is again based on how they handle uncertainty. The Frequentist estimator is a form of an expectation. As such it has nice properties when thinking in terms of truth or falsehood or more properly rejection of a null. On the other hand, the Bayesian solution actually creates an entire statistical distribution of possible answers. When you estimate the number of future events that could happen, a density function provides more information to the predictive estimate than a single point will, no matter how well chosen.

What has not been discussed are things such as logit, probit or related methods. The biggest issue there is that your number of factors have to be small relative to your observed defaults. Logit is the easier to discuss because it is directly related to the methods used above, except that the problem is converted into one of odds, to be more precise, log odds.

All the limitations seen above are present in logistic regression except that you are trying to condition the prediction on factors. The difficulty here is that you have few examples of an event and so while factors may be very good at explaining variation among the observed non-defaults, they may not be good at predicting the difference between defaults and non-defaults.

The form is restricted by the size of defaults and non-defaults. Imagine you were sure eight factors were involved in separating defaults from non-defaults, but you only saw nine examples of defaults. Further, you think one factor is actually quadratic. You will have an overfitting problem. The very thing you need, which is a highly predictive model for these rare events, is the very thing you cannot have. Fortunately, Bayesian model selection processes will provide a disciplined mechanism for you to make estimates with, but it would be very surprising to find the successful model had 8 factors with the above data.

On top of this, most defaults are not "all-or-nothing," and on top of this, you will need to model the actual amount at risk and not just the rate. This will reduce your independent variables down quite a bit.

If your useage is academic or existential, then use the Frequentist method. If this is being used for a business then the Bayesian method should be used.