1
$\begingroup$

I am trying to develop a trading model. It uses certain technical and fundamental features and the model learns from the past. I have a 3-class output - bullish, neutral and bearish.

On trying neural networks, I got a train accuracy of around 85%, cross validation accuracy of 75% and test accuracy of 75% with both CV set and test set carved out from the complete set. I am doing it on matlab 2010 using nprtool. The software does the carve-outs of the CV set and test set (20% each and the rest of 60% is used for training).

After training I use the same model to test on a different data. Here the accuracy drops to around 34% (which I would assume is just random classification with 3 classes). The new data I test with is very similar to the data used to train. I also tried swapping the data sets, but the results are same. I used the command below to test with the model (the same was used to test the accuracy of the train data too with Xtest replaced with Xtrain). Here net denotes the trained model.

output = sim(net, Xtest');

For NN I used 100,000 samples, 250 input features and 100 nodes in the hidden layer.

I tried a similar thing using libsvm (called from matlab). Here too I face the same. During training, the parameters are optimized through cross validation and the CV accuracy comes up to 73%. However when tested on new data the accuracy drops to around 35%.

For SVM I trained with 20,000 samples and 250 features. I use the command below:

[outputTest, accuracyTest, prob] = svmpredict(Ytest,Xtest,model);

Please help if someone has come across a similar issue and has a remedy.

Improve this question
$\endgroup$
2
  • $\begingroup$ How are your Matlab programs doing cross-validation on a time-series? You can't just do 10-fold CV (or whatever) as normal, since you shouldn't be training on future data and testing on past data. $\endgroup$
    – Chan-Ho Suh
    Dec 29, 2013 at 1:12
  • $\begingroup$ Yes, that is the major mistake I made. I just considered the time series as independent data points. $\endgroup$
    – dgmattam
    Dec 31, 2013 at 4:43

2 Answers 2

Reset to default
0
$\begingroup$

Firstly, look at Jurik Research WAV and DDR modules to see one particular approach for time series data compression and decorrelation prior to input into ANNs. From recent research, it also seems like better results have been obtained using wavelets on lagged (2,5,10 bar) data along with indicators and summary statistics such as skewness, kurtosis.

You have too many inputs, and if there is any correlation between them, you will be defeating the purpose of ANNs. A basic rule of ANNs is that they will waste time learning the correlation between input features, so if it is removed and orthogonal inputs (zero correlation) are used, the results may be better.

Last, intermarket analyses has also produced better results than staying within assets. That is, start looking at other indices (lagged and unlagged) as inputs.

Don't fret over poor results -- I have obtained similar results using normalized data, 4 wavelets representing highs and four wavelets representing lows (and a variety of lags) for a 2-class problem (tomorrow is a gain(y>0) or loss(y<0)) with very poor results.

Improve this answer
$\endgroup$
2
  • $\begingroup$ All valid inputs. However correlation between features shouldn't be such a major issue with SVM. I think the primary reason for it going wrong is what Chan-Ho Suh mentioned in his comment above. Also the cross validation accuracy is highly sensitive to the gaussian parameter, kind of reducing the generality of the model. Will try with different features as you suggested. $\endgroup$
    – dgmattam
    Dec 31, 2013 at 4:49
  • $\begingroup$ Here's the recent 2012 neuro-wavelet paper on stock time series data I was referring to. $\endgroup$
    – user6430
    Dec 31, 2013 at 5:09
0
$\begingroup$

There can be several reasons for this:

  1. The "new data" that you use post-training & post-validation is not drawn from the same distribution as the one that you used to create/draw your training, testing and validation data.
  2. Since you have not mentioned anything related to the input features in your data-set, I am assuming that the stock/option/derivative/instrument price is one of them. I also assume that you know that the price movement is stochastic in nature (or at least the current accepted model is that price movements exhibit a stochastic random-walk behavior). You need to account for this. You can take a look at any standard textbook on quantitative finance (e.g. Wilmott).
  3. You might be overfitting the model
  4. The curse of dimensionality.
  5. Your data is not normalized/preprocessed in the right way. Again, I can't say more as I don't know how your data looks like.
  6. In principle, it is a very HARD problem. If it was easy, well, all of us'd be retired. I mean there is no free-lunch. :)
Improve this answer
$\endgroup$
2
  • $\begingroup$ 1.It is drawn from the same distribution. I am using high frequency data; it is unlikely there is a major shift in a few days. $\endgroup$
    – dgmattam
    Dec 28, 2013 at 14:36
  • $\begingroup$ 2. Input features as of now do not include prices. 3. Over-fitting is a possibility, but cross validation should have helped to make the model more general by choosing damping factors . Either way, the mentioned test accuracy was for a fresh data not overlapping with the train data (I would presume NN in matlab does it that way). 4. That's a possibility as I have sparse arrays. 5. I've scaled the data by just normalizing it. $\endgroup$
    – dgmattam
    Dec 28, 2013 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.