# How to justify a model that could not predict external factors?

I'm building some models, for example, Bad Loan (NPL) rate.

It's based on historical simulation method -- basically it's saying the future behavior could be predicted by history data.

However, this is not always true, when the market changes. For example, when 2008 financial crisis came, NPL rates went up; when there's a tax cut, the NPL drops.

So, whenever market changes, the model will fail, but after a few months, it will pick up the trend and work again.

Now, how could I defend my model? How to justify that the failure is acceptable? Is there some industry standard / criterion about the tolerance of model mismatch caused by external factors (e.g. market, economics)?

Well, typically in the process of coming up with a model you are supposed to understand the assumptions that you're making and the circumstances(preferably quantifiable) under which your assumptions will hold/break. No model is infallible and it is how well the assumptions are stated and understood that will determine if your model is acceptable.

I can't really provide any specific suggestions since I don't know about the fine details of what you're doing. But I'd looking into applying some form of granger causality metric if time-series are involved or to measure the non-stationarity of the data(entropy and f-divergences).

• thx! first time heard about Granger causality... it smells nice! Mar 22, 2013 at 6:19

it was meant to be a comment, so please don't treat it like an answer, just suggestion.

I think every department has its own standards. and if you want to constitute your model somehow, then you can just compute R^2 between real and fitted values, RMSD, information capacity criteria (AIC/BIC) or you can use any from tens other measures. you can also state that there is some long-run relationship, and these deviations are just short term deviations from the long-run equilibrium, maybe ECM (error correction) model is appropriate.

• thanks for the reply. AIC/BIC are comparing among models, while I have only 1 here. I like the ECM idea, let me have a look.... Mar 22, 2013 at 2:55
• AIC, BIC ant rest are also used when optimizing model in MLE, so one can show that his model has nice AIC compared to other model Mar 22, 2013 at 18:21
• @athos you can also specify acceptable confidence interval and show that model is always within them, little trial an error and surely you will manage to fit your model to observed discrepancies Mar 22, 2013 at 20:52

I think most models failed in the 2008 crisis. Historical simulation and e.g. a value-at-risk calculated from it is designed for normal to "medium" market behavior.

To account for crisis scenarios stress tests should be in place. This what is done e.g. in the USCITS framework.

However, after the crisis your model should keep this history "in mind".

• I think most models failed after first use. (econometrics models to "predict" data - this is misunderstanding) Mar 22, 2013 at 18:22