I want to perform features selection on my dataset. I've split my data into train, test and out-of-sample set. The dataset is time-series based, so the split is sequenced in the order that train set will be taken from the earliest segment of the dataset and the out-of-sample set will be taken from the latest segment of the dataset.
I apply spearman ranking on all 3 sets to obtain corrcoef as well as filter for only features with pvalue < 0.02. I noticed that all 3 sets reported a very different sets of correlated features. And in some features, corrcoef maybe positive in the train set, but on the test set is negative.
Unsurprisingly any model trained using the train set and validated using the test set, most likely will perform miserably on the out-of-sample. I suppose this is a very common problem in a quant finance dataset. How can I solve this?