8
$\begingroup$

I'm building some models, for example, Bad Loan (NPL) rate.

It's based on historical simulation method -- basically it's saying the future behavior could be predicted by history data.

However, this is not always true, when the market changes. For example, when 2008 financial crisis came, NPL rates went up; when there's a tax cut, the NPL drops.

So, whenever market changes, the model will fail, but after a few months, it will pick up the trend and work again.

Now, how could I defend my model? How to justify that the failure is acceptable? Is there some industry standard / criterion about the tolerance of model mismatch caused by external factors (e.g. market, economics)?

|improve this question
$\endgroup$
3
$\begingroup$

Well, typically in the process of coming up with a model you are supposed to understand the assumptions that you're making and the circumstances(preferably quantifiable) under which your assumptions will hold/break. No model is infallible and it is how well the assumptions are stated and understood that will determine if your model is acceptable.

I can't really provide any specific suggestions since I don't know about the fine details of what you're doing. But I'd looking into applying some form of granger causality metric if time-series are involved or to measure the non-stationarity of the data(entropy and f-divergences).

|improve this answer
$\endgroup$
  • $\begingroup$ thx! first time heard about Granger causality... it smells nice! $\endgroup$ – athos Mar 22 '13 at 6:19
2
$\begingroup$

it was meant to be a comment, so please don't treat it like an answer, just suggestion.

I think every department has its own standards. and if you want to constitute your model somehow, then you can just compute R^2 between real and fitted values, RMSD, information capacity criteria (AIC/BIC) or you can use any from tens other measures. you can also state that there is some long-run relationship, and these deviations are just short term deviations from the long-run equilibrium, maybe ECM (error correction) model is appropriate.

|improve this answer
$\endgroup$
  • $\begingroup$ thanks for the reply. AIC/BIC are comparing among models, while I have only 1 here. I like the ECM idea, let me have a look.... $\endgroup$ – athos Mar 22 '13 at 2:55
  • $\begingroup$ AIC, BIC ant rest are also used when optimizing model in MLE, so one can show that his model has nice AIC compared to other model $\endgroup$ – 4pie0 Mar 22 '13 at 18:21
  • $\begingroup$ @athos you can also specify acceptable confidence interval and show that model is always within them, little trial an error and surely you will manage to fit your model to observed discrepancies $\endgroup$ – 4pie0 Mar 22 '13 at 20:52
1
$\begingroup$

I think most models failed in the 2008 crisis. Historical simulation and e.g. a value-at-risk calculated from it is designed for normal to "medium" market behavior.

To account for crisis scenarios stress tests should be in place. This what is done e.g. in the USCITS framework.

However, after the crisis your model should keep this history "in mind".

|improve this answer
$\endgroup$
  • $\begingroup$ I think most models failed after first use. (econometrics models to "predict" data - this is misunderstanding) $\endgroup$ – 4pie0 Mar 22 '13 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.