Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm building some models, for example, Bad Loan (NPL) rate.

It's based on historical simulation method -- basically it's saying the future behavior could be predicted by history data.

However, this is not always true, when the market changes. For example, when 2008 financial crisis came, NPL rates went up; when there's a tax cut, the NPL drops.

So, whenever market changes, the model will fail, but after a few months, it will pick up the trend and work again.

Now, how could I defend my model? How to justify that the failure is acceptable? Is there some industry standard / criterion about the tolerance of model mismatch caused by external factors (e.g. market, economics)?

share|improve this question
up vote 2 down vote accepted

Well, typically in the process of coming up with a model you are supposed to understand the assumptions that you're making and the circumstances(preferably quantifiable) under which your assumptions will hold/break. No model is infallible and it is how well the assumptions are stated and understood that will determine if your model is acceptable.

I can't really provide any specific suggestions since I don't know about the fine details of what you're doing. But I'd looking into applying some form of granger causality metric if time-series are involved or to measure the non-stationarity of the data(entropy and f-divergences).

share|improve this answer
thx! first time heard about Granger causality... it smells nice! – athos Mar 22 '13 at 6:19

it was meant to be a comment, so please don't treat it like an answer, just suggestion.

I think every department has its own standards. and if you want to constitute your model somehow, then you can just compute R^2 between real and fitted values, RMSD, information capacity criteria (AIC/BIC) or you can use any from tens other measures. you can also state that there is some long-run relationship, and these deviations are just short term deviations from the long-run equilibrium, maybe ECM (error correction) model is appropriate.

share|improve this answer
thanks for the reply. AIC/BIC are comparing among models, while I have only 1 here. I like the ECM idea, let me have a look.... – athos Mar 22 '13 at 2:55
AIC, BIC ant rest are also used when optimizing model in MLE, so one can show that his model has nice AIC compared to other model – where_is_tftp Mar 22 '13 at 18:21
@athos you can also specify acceptable confidence interval and show that model is always within them, little trial an error and surely you will manage to fit your model to observed discrepancies – where_is_tftp Mar 22 '13 at 20:52

I think most models failed in the 2008 crisis. Historical simulation and e.g. a value-at-risk calculated from it is designed for normal to "medium" market behavior.

To account for crisis scenarios stress tests should be in place. This what is done e.g. in the USCITS framework.

However, after the crisis your model should keep this history "in mind".

share|improve this answer
I think most models failed after first use. (econometrics models to "predict" data - this is misunderstanding) – where_is_tftp Mar 22 '13 at 18:22

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.