If a random variable is discrete, and we are interested in its quantile value, how to define a proper back testing procedure?
For example, the underlying variable with a discrete value is
$$ d(\mbox{account}) = \mbox{PaymentDate} - \mbox{BillingDate} $$
the observing variable:
$$ y = \mbox{percentile}(d, 95\%, \mbox{month}) $$
or $y$ is the 95th percentile value of $d$, for a particular month. e.g. 95% of credit cards are paid within 20 days from the billing, in 2013 Jan.
How could I define a back-testing approach?
Background
To define an estimation-backtesting method for a continous random variable is easier. Now in my group we have such a non-parametric approach:
underlying variable:
$$r(\mbox{month}) = \mbox{monthly credit-card account default rate}$$
For example, 2013 Feb default rate is 1.1%, 2013 Jan is 1.2%...
observing variable:
$$ x = \mbox{percentile}(r, 95\%) $$
$x$ is the 95% percentile value of $r$. Here $x$ definition is similar to VaR.
point forcast:
$$ \hat x(\mbox{month}) = \mbox{percentile}(r(\mbox{month}), N, 95\%) $$
$\hat x$ is the 95% percentile value of $r$, based on $N$ historic observations.
For example, take $N=36$, retrieve back 36 months, the 95% percentile value of default rate $r$ is 2.3%. then $\hat x = 2.3\%$.
point forecast Exception:
$$ \mbox{PFException}(t) = \begin{cases} 0 & r(\mbox{month}) \leq \hat x(\mbox{month}) \\ 1 & \text{otherwise} \end{cases} $$
By right 95% of the time there shall have no exception, while 5% of the time exception happens.
backtesting:
There are POF test, checking the rate of the exception; and independent test, checking the correlation of exceptions.
For example, Kupiec (1995) proposed a POF test checks exceptions happened in previouis 36 months' point forecasts: 0-4 exceptions are ok, green light, 4-7 exceptions are yellow light, while more than 8 exceptions are red light.
Christoffersen (1998) proposed an independent test.
Kupiec, P. (1995). Techniques for verifying the accuracy of risk management models. Journal of Derivatives 3, 73–84.
Christoffersen, P. (1998). Evaluating interval forecasts. International Economic Review 39, 841–62.
