0
$\begingroup$

This question relates to the use of random forests in finance and the relationship between the number of features, the observations, and the number of trees.

Consider the relation between an RF, the number of trees it is composed of, and the number of features utilized:

  • Could you envision a relation between the minimum number of trees needed in an RF and the number of features utilized?
  • Could the number of trees be too small for the number of features used?
  • Could the number of trees be too high for the number of observations available?

Question sourced from (AFML de Prado 2018)

$\endgroup$
1
$\begingroup$

The following post on cross-validated has quite a good answer:

"Random forest uses bagging (picking a sample of observations rather than all of them) and random subspace method (picking a sample of features rather than all of them, in other words - attribute bagging) to grow a tree. If the number of observations is large, but the number of trees is too small, then some observations will be predicted only once or even not at all. If the number of predictors is large but the number of trees is too small, then some features can (theoretically) be missed in all subspaces used. Both cases results in the decrease of random forest predictive power. But the last is a rather extreme case, since the selection of subspace is performed at each node."

I ran some empirical tests to validate this. First I created synthetic data using the following:

# Create data
X, y = make_classification(n_samples=20000, n_features=50,
                            n_informative=10, n_redundant=0,
                            random_state=42, shuffle=True, n_classes=2, class_sep=1.0)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, shuffle=True, stratify=None)

There are 50 features, with only 10 of them being informative. There is a clear class separation and 20000 observations.

Next, I fit a random forest which is limited by the number of trees, and the number of features it may use (n_estimators, max_features).

max_trees = 100
max_feat_used = 50

store = []
for num_trees in range(2, max_trees, 2):
    print(num_trees)
    for num_feat in range(1, max_feat_used, 2):
        rnd_clf = RandomForestClassifier(criterion='entropy', n_estimators=num_trees, max_features=num_feat, n_jobs=-1)
        rnd_clf.fit(X_train, y_train) 
        y_pred_rf = rnd_clf.predict(X_test)

        store.append([num_trees, num_feat, accuracy_score(y_test, y_pred_rf)])

# Pivot and save results
results = pd.DataFrame(store, columns=['N', 'F', 'Score'])
pivot_results = results.pivot(index='N', columns='F', values='Score')
pivot_results = pivot_results.sort_index(ascending=False)

Finally, we can observe the relationship in a heat map:

Relationship

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.