Missing factor in the factor model

I am developing a factor model to predict monthly returns. One of the factors alone accounts for an R squared of 0.3 to 0.4 for many single periods that has surprised me.

However, for some periods the direction of the factor reverse completely and if I run a single regression for all the periods this factor alone accounts for an R squared of 0.1 which means that the factor is robust but I am missing another factor, correct me if there is another explanation.

Does anyone have any opinion on how to isolate the explanatory power of this factor in the panels in order to eliminate the effect of missing factor or any analytical way to flag the other factor?

-
There is not enough information for me to answer it but - your statement One of the factors alone accounts for an R squared of 0.3 to 0.4 for many single periods that has surprised me. Statistically it is not surprising and I need to know more about the factor and what you are modelling and your assumptions are to pin point if there is anything wrong with it If you think it is important factor than have it by all means. See the outliers and try to analyse them. –  ash Nov 16 '12 at 9:38

Your data also shows signs of a time-varying correlation with your factor. Hence, it $may$ also be appropriate to allow for time-varying regression coefficients or some other technique to account for this feature of the data.