1
$\begingroup$

I have never really given thought to this, but training some strategies I'm testing today I noticed that my model returns an acceptable annualized return/drawdown/etc, but the model parameters are not significant (according to p-value).

To me, this tells me that these model parameters do not adequately explain the response variable's value. But if this was the case, I would expect to see very poor backtesting/walk-forward/etc results.

What context should I be viewing the p-value of parameters of a logistic regression in the context of finance?

$\endgroup$
  • $\begingroup$ basically your good results are due to luck $\endgroup$ – vonjd Mar 15 '18 at 7:59
  • $\begingroup$ @vonjd I find it hard to attribute an accurate model to pure luck. There has to be some underlying process that is allowing the model to predict accurately. Backtesting over 5 years of data with rolling windows should've shown some really unacceptable downturns if it was just luck. $\endgroup$ – user20664 Mar 15 '18 at 8:05
  • $\begingroup$ Thank you for accepting my answer. On top of that your high p-values are an additional warning sign. $\endgroup$ – vonjd Mar 15 '18 at 8:30
  • 1
    $\begingroup$ My thoughts on the econometric reason this model may work is that there is a high degree of autocorrelation in some of the early lags. The other questions I'll have to dig into after I read the paper. Thank you $\endgroup$ – user20664 Mar 15 '18 at 8:32
  • $\begingroup$ This econometric reasoning may warrant a follow-up question. Stylized facts are that there is (almost) no autocorrelation in the returns but significant autocorrelation in higher moments (e.g. volatility clustering). $\endgroup$ – vonjd Mar 15 '18 at 8:36
0
$\begingroup$

You state that you are testing models and not a model. The standard, built-in, statistical tests of significance presume you are testing only one model. The only general exception to this is using some solution such as stepwise regression which is built to allow for multiple comparisons. Procedurally, you should perform some variant of either the AIC or the BIC across your models. After this one model is selected, you would then perform a master F test. If the F-test were significant, then you would perform t-tests on each parameter. In stepwise regression, this all happens in one giant output. After you are satisfied with that, then you would perform out-of-sample validation.

If you don't do that, then your p-values don't mean anything. The AIC and BIC, and there are a few others, are actually probability statements. They are proportionate to the Bayesian posterior probability that the model is the true model under certain very stylized assumptions. You may hear the argument that this is not true for the AIC, and under certain axiomatic constructions that would be correct, but under Bayesian rather than information theoretic axioms it is just a non-normalized approximation of the Bayesian posterior.

The reason this is important is that the information criterion is the control for multiple comparisons. If you do not use this, then you would need to have a giant grand model of models with corrections for multiple comparisons.

What do you do with all of the other models? You discard them unless the information criterion is very close. Close is very small considering that these are logarithmic comparisons. Close is very very close, well under one unit apart.

Once you have chosen one model, then you look at the F-test. If it is not significant, then you discard that model too. You would then look for different variables to build models from.

If you F-test is significant then you look at your t-tests. In your case, there are no significant variables. There are several possible explanations.

The first one is that nothing is significant. There are a large number of chance correlations in finance because financial data are constructed in such a way as there cannot be independence. You can have correlations that are real, but also useless in the sense that they have no predictive value at all.

A second possibility is multicollinearity. This is a serious problem because your information criterion felt this was the best model. If it is multicollinearity then you need to figure out which variables move close together and remove as many as possible. Principal components analysis can help here. Once you have reduced your variable set, start the process all over again with that reduced set.

If you had some variables that were significant and some that were not, then you cannot drop the ones that are not significant if the subsetting models had been considered under the information criterion. For example, consider $y=\alpha+\beta_1x_1+\beta_2x_2$ This model was accepted under the information criterion and the models where either $x_1$ or $x_2$ were omitted were rejected. The information criterion implies the data is necessary for a good model, but you lack sufficient power to falsify the null. It may be the case that $\beta_2$ is not zero but is very close to zero.

After you have completed all of this, you can start with your out-of-sample validation. If you do not like the validation results, start over with new types of data.

The key element of this is the information criterion. They are non-Frequentist estimators and so are not bound to the null hypothesis problem. Bayesian probability tests do not reject a null. There is no special hypothesis in Bayesian thinking. Bayesian hypotheses test if it is true and do not test whether a null is false. As a consequence, you should have as many hypotheses as you have options available.

Just so you have a feel for the differences, I am attaching a link from MIT's OpenCourseWare. Differences between types of inference. There is also a decent paper by Wagenmakers comparing t-tests with Bayesian posterior densities over 855 published t-tests in the existing literature. It is at Wagenmakers website. You should really read it to get a better feel for what a t-test is and is not.

$\endgroup$
  • $\begingroup$ This is fascinating - I have accepted this as the answer. One question though: in the case of testing the predictive power of a model, would you even need to consider AIC/BIC/etc, or would a "hit rate" and confusion matrix be sufficient? $\endgroup$ – user20664 Mar 17 '18 at 19:11
  • $\begingroup$ I was working on the assumption that you were using this for prediction. If you have multiple models, you always begin with the information criterion. If you inspect the outcomes closely, they won't usually give you the best in-sample results. Bayesian methods attempt to solve what the data generating function is rather than just the parameters when multiple models are present. This method approximates the more rigorous Bayesian solution. You want to ignore your validation work until last. Otherwise, you have a very good chance to overfit. $\endgroup$ – Dave Harris Mar 17 '18 at 21:26
1
$\begingroup$

Basically there are at least two questions you have to ask yourself before believing "good backtests":

  1. Is there an economic or behavioural explanation why this should work (best would be to start from there in the first place).
  2. How many things did I try to arrive at this "good backtest".

The following paper is quite helpful to learn some of the basics of this kind of thinking:

Bailey, David H. and Borwein, Jonathan and Lopez de Prado, Marcos and Zhu, Qiji Jim, Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance (April 1, 2014). Notices of the American Mathematical Society, 61(5), May 2014, pp.458-471. Available at SSRN: https://ssrn.com/abstract=2308659 or http://dx.doi.org/10.2139/ssrn.2308659

Abstract
We prove that high simulated performance is easily achievable after backtesting a relatively small number of alternative strategy configurations, a practice we denote “backtest overfitting”. The higher the number of configurations tried, the greater is the probability that the backtest is overfit. Because most financial analysts and academics rarely report the number of configurations tried for a given backtest, investors cannot evaluate the degree of overfitting in most investment proposals.

The implication is that investors can be easily misled into allocating capital to strategies that appear to be mathematically sound and empirically supported by an outstanding backtest. Under memory effects, backtest overfitting leads to negative expected returns out-of-sample, rather than zero performance. This may be one of several reasons why so many quantitative funds appear to fail.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy