Steps to fit a Machine learning model for prediction of up and down market movement

Question

I have around 5 years of data of an index containing many features on a daily basis. I want to classify whether the index will move up or down the next trading day (up or down movement is determined by next day open/close price). I am using an SVM classifier for this classification. What could be some essential steps which need to be followed? I suppose since I am using financial data, there would be some deviation from the traditional method of applying machine learning. I have the following steps in mind:

Prepare data containing all features at-hand and direction variable.

Feature engineering: That is creating various features using transformations like log, % change on various available features. I have not incorporated this step but will do it later when I have a decent running model.

Feature Selection: How to go about this? I first am using various features which I think have predictive power but how to go about it systematically.

Walk forward Modelling: After selecting features I have the data which I will be using for my model. I am using the first 500 days of data to train the model and testing it on the next day and then using data from day 2 to day 501 to train the model and then testing it for day 502, and so on until I reach the present day. I make sure that I scale my training data in the range 0-1 and using the same scaler for test data before running the model. I use the default parameters of SVM in sklearn right now.

I then check the performance of my model by using various metrics obtained from confusion matrix-like accuracy, F1 score, etc.

For applying SVM I am using this reference which tells to use cross-validation to choose C and gamma. How can we use CV here if we have financial time series data? What else should I include in my steps?

Answer 1

For true cross-validation you need $N$ separate calibration sets, for example one data set per year. You can then test performance on, say, the month of data following each calibration set.

In feature engineering (FE), the essential problem is that if you systematize FE to "try many things", it becomes much harder to avoid the pernicious issues of overfit that occur in high-noise situations like financial return predictions. In my opinion you are better off avoiding engineering until after you have satisfied yourself that what you attempt is even possible.

Your conception of "walk-forward modeling" is reasonable and important, so kudos to you on that score. Note that training data scaling needs to be done in the same manner, lest you introduce one of the many types of "look-ahead bias".

Needless to say, many brilliant people have pursued SVMs for this kind of prediction before you, and have arbitraged away any profits they found. Unless your data set contains unusual or private categories of data, I recommend you view this project as an intellectual/academic exercise more than a profit opportunity.