I am trying to develop a trading model. It uses certain technical and fundamental features and the model learns from the past. I have a 3-class output - bullish, neutral and bearish.
On trying neural networks, I got a train accuracy of around 85%, cross validation accuracy of 75% and test accuracy of 75% with both CV set and test set carved out from the complete set. I am doing it on matlab 2010 using nprtool. The software does the carve-outs of the CV set and test set (20% each and the rest of 60% is used for training).
After training I use the same model to test on a different data. Here the accuracy drops to around 34% (which I would assume is just random classification with 3 classes). The new data I test with is very similar to the data used to train. I also tried swapping the data sets, but the results are same. I used the command below to test with the model (the same was used to test the accuracy of the train data too with Xtest replaced with Xtrain). Here net denotes the trained model.
output = sim(net, Xtest');
For NN I used 100,000 samples, 250 input features and 100 nodes in the hidden layer.
I tried a similar thing using libsvm (called from matlab). Here too I face the same. During training, the parameters are optimized through cross validation and the CV accuracy comes up to 73%. However when tested on new data the accuracy drops to around 35%.
For SVM I trained with 20,000 samples and 250 features. I use the command below:
[outputTest, accuracyTest, prob] = svmpredict(Ytest,Xtest,model);
Please help if someone has come across a similar issue and has a remedy.