Take the 2-minute tour ×
Quantitative Finance Stack Exchange is a question and answer site for finance professionals and academics. It's 100% free, no registration required.

I'm building some models, for example, Bad Loan (NPL) rate.

It's based on historical simulation method -- basically it's saying the future behavior could be predicted by history data.

However, this is not always true, when the market changes. For example, when 2008 financial crisis came, NPL rates went up; when there's a tax cut, the NPL drops.

So, whenever market changes, the model will fail, but after a few months, it will pick up the trend and work again.

Now, how could I defend my model? How to justify that the failure is acceptable? Is there some industry standard / criterion about the tolerance of model mismatch caused by external factors (e.g. market, economics)?

share|improve this question
add comment

3 Answers 3

up vote 2 down vote accepted

Well, typically in the process of coming up with a model you are supposed to understand the assumptions that you're making and the circumstances(preferably quantifiable) under which your assumptions will hold/break. No model is infallible and it is how well the assumptions are stated and understood that will determine if your model is acceptable.

I can't really provide any specific suggestions since I don't know about the fine details of what you're doing. But I'd looking into applying some form of granger causality metric if time-series are involved or to measure the non-stationarity of the data(entropy and f-divergences).

share|improve this answer
    
thx! first time heard about Granger causality... it smells nice! –  athos Mar 22 '13 at 6:19
add comment

it was meant to be a comment, so please don't treat it like an answer, just suggestion.


I think every department has its own standards. and if you want to constitute your model somehow, then you can just compute R^2 between real and fitted values, RMSD, information capacity criteria (AIC/BIC) or you can use any from tens other measures. you can also state that there is some long-run relationship, and these deviations are just short term deviations from the long-run equilibrium, maybe ECM (error correction) model is appropriate.

share|improve this answer
    
thanks for the reply. AIC/BIC are comparing among models, while I have only 1 here. I like the ECM idea, let me have a look.... –  athos Mar 22 '13 at 2:55
    
AIC, BIC ant rest are also used when optimizing model in MLE, so one can show that his model has nice AIC compared to other model –  bits_international Mar 22 '13 at 18:21
    
@athos you can also specify acceptable confidence interval and show that model is always within them, little trial an error and surely you will manage to fit your model to observed discrepancies –  bits_international Mar 22 '13 at 20:52
add comment

I think most models failed in the 2008 crisis. Historical simulation and e.g. a value-at-risk calculated from it is designed for normal to "medium" market behavior.

To account for crisis scenarios stress tests should be in place. This what is done e.g. in the USCITS framework.

However, after the crisis your model should keep this history "in mind".

share|improve this answer
    
I think most models failed after first use. (econometrics models to "predict" data - this is misunderstanding) –  bits_international Mar 22 '13 at 18:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.