Rating systems, as defined by the Basel II Accord, can be classified into two broad types - through-the-cycle (TTC) or point-in-time (PIT) - and the probability of default predicted by such a system can usually have different interpretations. In practice, no rating system is purely TTC or PIT.
One advantage of PIT systems are that they are easy to 'backtest' - if the PIT system predicts the 1 year probability of default, we can always "score" the obligors using the system 1 year back, and then compare the actual default rates against the predicted probability of default.
Since TTC systems predict the probability of default over different economic cycles, it is hard to backtest such a system as the realized default rates may not match with the probability depending on where we are in the economic cycle.
I am having a hard time finding any existing research or literature on this topic. Most of them mention the standard measures like ROC, Gini, Accuracy Ratio, etc. which tells us whether the rating system can sufficiently differentiate the likelihood of default. However, this does not say if the probabilities output by the rating system make sense.
Can anyone point me out to any relevant research, books or documents which may help?