Choosing attributes for SVM classification?

Question

Let's assume I am classifying every trading day as a 1 or a 0. Exactly what I am classifying doesn't matter, but for the sake of this question let's say I am predicting direction of price change. So, for a particular stock like GOOG:

For day N:
1: close day N+1 > close day N
0: close day N+1 <= close day N

For every day N, I want to predict whether it is a 1 or 0.

Now let's assume that, as input, I want to use a time series (previous 10 days) of:

1. Closing price
2. EMA closing price
3. Intra-day volatility

That will give me 30 attributes as input. My question, is what makes more sense:

• Generate three different classification models (one for each of the above attribute types). Then perform one last classification which will give me a final prediction, which takes the predicted class of each model as input. I.E. This final classification will contain three attributes as input, like [1,0,1] or [1,1,1] etc.

• Perform one classification which has all 30 attributes as input.

Answer 1

Use all the attributes in a single model.

If you build three separate models, you will be throwing away all all information that might be contained in combinations of different features.

So, for example, it might be the case that prices are more likely to go up tomorrow if today's closing price was above the ema and volatility was high, but it is more likely to go down with the same price action if volatility was low. The individual models wouldn't be able to capture that correlation, because no single SVM would be able to see all the required data, but a single SVM with all the attributes could detect it (with a non-linear kernel).

Answer 2

You are confronted here in the common "bottom-up" or "top-down" problem.

I think there is no final answer to your question, as both approach have their pros and cons.

For the "bottom-up", you first classify for each feature, then classify again. This give you the ability to get a better understanding of the decision of your algorithm by splitting the decision in 2 steps. However, as Marc Shivers explains in his answer, you will lose part of the precision the method could have once you have completed step one, and hence your algorithm can be less precise.

For the "top-down", you straightaway run the SVM over the full set of parameters. The first advantage is that it is simpler, and it is the plain-vanilla way of applying the algorithm. A problem is that the algorithm might be tempted to overfit the data, i.e. to give you a solution that is "too precise", and hence wrong.

Choosing between underfitting or overfitting is very difficult. You should try both methods and really assess which ones performs the best in the conditions you are considering. Indeed, the performance of a method can vary depending on the amount of data you have and other characteristic which are difficult to guess a priori. Andrew Ng provides a fantastic online class on coursera where you can see different ways of evaluation your algorithm in chapter 10.