Testing the validity of a factor model for stock returns

Question

Consider the following m regression equation system:

$$r^i = X^i \beta^i + \epsilon^i \;\;\; \text{for} \;i=1,2,3,..,n$$

where $r^i$ is a $(T\times 1)$ vector of the T observations of the dependent variable, $X^i$ is a $(T\times k)$ matrix of independent variables, $\beta^i$ is a $(k\times1)$ vector of the regression coefficients and $\epsilon^i$ is the vector of errors for the $T$ observations of the $i^{th}$ regression.

My question is: in order to test the validity of this model for stock returns (i.e. the inclusion of those explanatory variables) using AIC or BIC criterion, should these criterion be computed on a time-series basis (i.e. for each stock), or on a cross-sectional basis (and then averaged over time)?

Answer 1

I would like to point out a recent paper by Lewellen, Nagel and Shaken which has changed a little bit the way factor models are tested. The standard procedure was to run time series regression of a factor model on Fama&French 25 size and BE/ME sorted portfolios to obtain factor loadings, and then cross sectional regressions using $R^2$ as a good measure for the goodness of the model. What Lewellen, Nagel and Shanken showed is that - given the great explanatory power of Size and Value - whenever we test a factor which is a little bit correlated with HML and SMB it’s very easy to get high $R^2$.

So they suggest a new procedure to test factor models, which is becoming the standard in Asset Pricing literature:

-test the model using a two steps procedure (e.g. Fama-Macbeth) for many portfolios other than 25 size and BE/ME sorted ones. Some examples could be industry, volatility, beta sorted portfolios and also bond portfolios.

-compare the values for cross-sectional sloped with the ones predicted by theory

-use GLS cross-sectional $R^2$

-whenever the factor is traded, include it in the regression and check whether it is correctly priced. Alternatively construct a “factor mimicking portfolio” and then test whether it is efficient

-check confidence intervals both for $R^2$ and Total Sum of Squared Errors

As far as I know very few models pass the Lewellen, Nagel and Shanken test and I bet if you want to check whether your model is good you should go to this route and see what are the results.

Answer 2

This answer depends on the $X^i$

Before jumping on to the solution it should be answered that are $X^i$ traded in the market? i.e. are the returns on these available in the market (Size/Momentum portfolios, ETF returns) or are these economic variables like CPI, Inflation etc.

If it is the former i.e. traded assets then we can do the time series regression to compute the factor loadings i.e. $\beta_i$. however we need to ensure other things while performing the regression, like multi-collinearity check etc. to avoid spurious results. People have used PCA and Clustering techniques to check for these things. Example, Fama-French model, Cahart 4 factor model.

For using economic variables like CPI, inflation, unemployment data we will need to do the cross-sectional regression since we don't know what is the factor risk premium i.e. $$E[R^i - rf]$$

So this will be a two step regression. You can use Fama-Macbeth or a similar procedure. Example, Chen-Ross-Roll model.

Answer 3

For calculating the AIC for factor models, I calculate the likelihood based on the multivariate distribution of the factor model. I try to make any assumptions as explicit as possible. Bayesians typically do not use the (so-called) BIC. WAIC (Watanabe-Akaike Information Criteria) is becoming more common among Bayesians.

When thinking about the AIC, you should try to clarify what you are using it for. More often than not it is optimization or risk management. I'm more familiar with optimization, so I'll try to keep my observations constrained to that area. For the sake of simplicity, I'll assume a multivariate normal here, but this can easily be relaxed.

Either a time series or cross-sectional model can be thought of as a multivariate distribution. For instance, in the time series approach, you would just make some explicit assumption about the correlations between stocks. In a rolling cross-sectional approach, you can use the mean and covariance of the coefficient/factor returns and then get a conditional multivariate distribution (i.e. it changes each period depending on the factor exposures).

There are two clear ways to construct the multivariate distribution (regardless of time series or cross-sectional). Either using the mean and covariance of the factors to get the mean and covariance of the stocks in the universe. Alternately, you treat the factors as exogenous and don't bother with the mean and covariance of the factors (at the model evaluation stage).

You will get different log likelihoods in each case. I prefer the first approach for a few reasons. The first is that it is more sensible to compare against a base case of a multivariate normal. Second, it is more consistent with what is important for portfolio optimization (the mean and covariance of each stock). Third, using market returns will necessarily make your error smaller and make your model look better (when it may not actually be). Fourth, factor returns are often not exogenous. We still have to forecast them for portfolio optimization. Finally, I get really confused about what is an isn't a parameter for AIC (market weighted returns are the mean of a cross-sectional distribution, what really makes them different than the regression coefficients from a cross-sectional regression). I'd rather abstract away from it and focus on the mean and covariance of the factors, which I can be pretty confident are parameters that matter for AIC.

I'm not saying you can't take the other approach, nor can I recall any academic research that says not to do it one way versus the other. I might be more inclined to use the other approach when I can't write down the likelihood function (i.e. for more sophisticated models).

While AIC can be useful in factor modelling, such as answering questions like should I use a market weighted factor or an equal weighted factor, I don't put a heavy emphasis on it personally. It's definitely not what I start with. My typical approach begins with some baseline model and proceeds by evaluating (quantitatively or qualitatively) whether there are some features of the data that the model is not capturing.

Bayesians call this a posterior predictive check. The goal of posterior predictive checks is to evaluate in what ways your model does not fit the data. What is the model not capturing. To perform this check, simulate returns from the model and see if they produce patterns that match actual returns. Do the simulated mean, standard deviation, and correlations match what you would expect? How does a subset look (just focusing on country or sector or large/small or value/growth differences)?