14
$\begingroup$

I'm using around 130 technical indicators for 100 different companies. Each company's stock price moves in a different range, see FTSE 100. In addition, each technical indicator moves in a different range as well, ie some goes between 0-1, other 0-100 and others move with the price. This is the list I'm using http://ta-lib.org/function.html

I'd like to feed this into a machine learning algorithm where I predict the relative price movement of the stock price the next day. I use a logistic loss for profit optimization and two regularizer terms, one between companies, and the other between time periods. This is unimportant for now.

Rather, what I'm asking about is how to normalize the input data? I've tried zscore, rate of change, absolute differences, and various combination of these, but I'm not sure which is the right approach. Also, I assume that first I need to calculate the indicators and then normalize that data. or would these indicators make sense on normalized data?

$\endgroup$
  • 4
    $\begingroup$ I'm not sure there is one right approach. $\endgroup$ – John May 7 '13 at 14:34
  • $\begingroup$ @John what would be a right approach? $\endgroup$ – siamii May 7 '13 at 14:36
  • $\begingroup$ My opinion: PCA over these indicators after having z-scaled observations to make covariance matrix equal to correlation matrix. $\endgroup$ – Lisa Ann May 8 '13 at 8:56
  • 1
    $\begingroup$ There will be too much correlation (multicollinearity) between all of your indicators. You could reduce the dimensions using PCA, but that's a linear technique. There may be better payoff using non-linear dimension reduction techniques (diffusion maps, Laplacian eigenmaps, locally preserving projections, Sammon's mapping, self-organizing maps). I have found that among 9 non-linear methods, SOM works the best for classification analysis. However, sounds like you are doing function approximation. $\endgroup$ – user5626 Jul 7 '13 at 17:13
  • $\begingroup$ What I do is put all indicators in bins. Some like volume have to be min(log(vol),some max), otherwise one high value will make bins that are too large. RSI types are easy as you know the max and min beforehand. For the rest you have to choose an appropriate max. $\endgroup$ – brian Aug 6 '13 at 14:39
3
$\begingroup$

There is no right approach a priori. Try all approaches that make decent sense and pick the one that maximises a utility function on out of sample PnL and risk (or some similar decision rule).

$\endgroup$
2
$\begingroup$

Scale and range are your biggest issues. If one input has values which range from e.g. 2300-3500, and another from 0 to 18, then the large scale of the first will swamp the other and provide greater informativity into your learning algorithm. Therefore, normalize into range [0,1] or mean-zero standardize - like you have already done. Be careful with mean-zero standardization, however, since means only apply to skew-zero normal distributions, and not to log-normally distributed (right-tailed) skewed distributions. You could input the ranks of your skewed features values, but ranks are known to be rectangularly distributed, so convert ranks to percentiles, and then to van der Waerden scores (which are standard normal distributed without skewness).

$\endgroup$
0
$\begingroup$

This depends on what ML you are using. For example, random forest is extremely resilient and doesn't require much normalisation of inputs to produce good outputs. However, RNN is extremely sensitive to input normalisation.

What I do for RNN is to divide each sample input by 1 standard deviation of the entire sample set, and that works quite nicely. But make sure your samples are normally distributed first.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.