Gradient tree boosting -- do input attributes need to be scaled?

Question

For other algorithms (like support vector machines), it is recommended that input attributes are scaled in some way (for example put everything on a [0,1] scale). I have googled extensively and can't find any information on whether this needs to be done for boosting methods, and in particular gradient tree boosting.

Answer 1

do input attributes need to be scaled?

No. It is not required. It is only a heuristic [1]. It is primarily motivated because of the following:

1. From the Feature Scaling article:

   Since the range of values of raw data varies widely, in some machine learning algorithms, objective functions will not work properly without normalization. For example, the majority of classifiers calculate the distance between two points by the distance. If one of the features has a broad range of values, the distance will be governed by this particular feature. Therefore, the range of all features should be normalized so that each feature contributes approximately proportionately to the final distance.

In summary,

• The recommendation for other algorithms like SVM is just 'recommendation'. It does not guarantee improved performance for instance.
• My suggestion is if this step is expensive, skip it. If it is not, then check to see if normalization does not deteriorate performance compared to building a Gradient boosted tree directly and decide accordingly.

Answer 2

For class prediction (vs. class discovery), I always transform (remove scale effects) for input features. Metrics for features with larger scale (range) can swamp metrics of features with smaller scale. Depending on your method (there are numerous boosting approaches), you might also consider decorrelating features. A golden rule regarding correlated features, at least with artificial neural networks, is that during connection weight learning time will be spent on figuring out the correlation -- herein lies the reason why many use linear dimensional reduction (e.g., PCA) prior to clamping objects to the input network nodes.

Overall, you should probably do both (with and without). And recall, there are choices for scaling: (a) you can remove skewness (fat-tails), (b) mean-zero standardize, and (c) normalize in range [0,1], (d) determine direction cosines of the input features so feature values are within a unit circle.

Last, never use a single classifier unless you need to. If possible, use an ensemble of classifiers so that one classifier can pick up on problematic objects when another breaks down. The No Free Lunch theorem states that in the universe of all cost functions, no one classifier is better than another - so never assume you have the best solution when using a single classifier. The Ugly Ducking theorem states that no one set of features is better than another -- so never assume your selected feature set is the best.