16
$\begingroup$

I'm using around 130 technical indicators for 100 different companies. Each company's stock price moves in a different range, see FTSE 100. In addition, each technical indicator moves in a different range as well, ie some goes between 0-1, other 0-100 and others move with the price. This is the list I'm using http://ta-lib.org/function.html

I'd like to feed this into a machine learning algorithm where I predict the relative price movement of the stock price the next day. I use a logistic loss for profit optimization and two regularizer terms, one between companies, and the other between time periods. This is unimportant for now.

Rather, what I'm asking about is how to normalize the input data? I've tried zscore, rate of change, absolute differences, and various combination of these, but I'm not sure which is the right approach. Also, I assume that first I need to calculate the indicators and then normalize that data. or would these indicators make sense on normalized data?

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ I'm not sure there is one right approach. $\endgroup$
    – John
    Commented May 7, 2013 at 14:34
  • $\begingroup$ @John what would be a right approach? $\endgroup$
    – siamii
    Commented May 7, 2013 at 14:36
  • $\begingroup$ My opinion: PCA over these indicators after having z-scaled observations to make covariance matrix equal to correlation matrix. $\endgroup$
    – Lisa Ann
    Commented May 8, 2013 at 8:56
  • 2
    $\begingroup$ There will be too much correlation (multicollinearity) between all of your indicators. You could reduce the dimensions using PCA, but that's a linear technique. There may be better payoff using non-linear dimension reduction techniques (diffusion maps, Laplacian eigenmaps, locally preserving projections, Sammon's mapping, self-organizing maps). I have found that among 9 non-linear methods, SOM works the best for classification analysis. However, sounds like you are doing function approximation. $\endgroup$
    – user5626
    Commented Jul 7, 2013 at 17:13
  • 2
    $\begingroup$ What I do is put all indicators in bins. Some like volume have to be min(log(vol),some max), otherwise one high value will make bins that are too large. RSI types are easy as you know the max and min beforehand. For the rest you have to choose an appropriate max. $\endgroup$
    – brian
    Commented Aug 6, 2013 at 14:39

4 Answers 4

Reset to default
3
$\begingroup$

Scale and range are your biggest issues. If one input has values which range from e.g. 2300-3500, and another from 0 to 18, then the large scale of the first will swamp the other and provide greater informativity into your learning algorithm. Therefore, normalize into range [0,1] or mean-zero standardize - like you have already done. Be careful with mean-zero standardization, however, since means only apply to skew-zero normal distributions, and not to log-normally distributed (right-tailed) skewed distributions. You could input the ranks of your skewed features values, but ranks are known to be rectangularly distributed, so convert ranks to percentiles, and then to van der Waerden scores (which are standard normal distributed without skewness).

Improve this answer
$\endgroup$
2
$\begingroup$

There is no right approach a priori. Try all approaches that make decent sense and pick the one that maximises a utility function on out of sample PnL and risk (or some similar decision rule).

Improve this answer
$\endgroup$
2
$\begingroup$

This depends on what ML you are using. For example, random forest is extremely resilient and doesn't require much normalisation of inputs to produce good outputs. However, RNN is extremely sensitive to input normalisation.

What I do for RNN is to divide each sample input by 1 standard deviation of the entire sample set, and that works quite nicely. But make sure your samples are normally distributed first.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ I think this add a lookahead bias to your data $\endgroup$
    – Alon Gouldman
    Commented Jun 9, 2020 at 12:43
1
$\begingroup$

There are a couple of good papers on this but my favorite are Predicting stock market index using fusion of machine learning techniques and Predicting stock and stock price index movement using Trend Deterministic Data Preparation and machine learning techniques.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.