Many small trees, and an honest ranking of features
A decision tree splits the rows again and again with a question like 'is the 20-day return above 1%?', choosing at each step the split that makes the groups purest. The Gini impurity of a group is G=1-Σₖpₖ², and a split is good when it lowers the weighted impurity of its two children. A tree of unlimited depth ends with one row per leaf: it reaches a train accuracy of 100% and a test accuracy near 50%. Two ways keep trees honest. A random forest averages many shallow trees, each grown on a random sample of rows and features, so their mistakes partly cancel. Gradient boosting adds small trees one after another, each fitted to what the sum so far still gets wrong: Fₘ=Fₘ₋₁+νhₘ, where the learning rate ν keeps every step small. Both are controlled by max_depth, min_samples_leaf and the number of trees, and both need random_state for repeatable results.
Feature importance comes in two kinds. The impurity importance (feature_importances_) adds up how much each feature lowered the impurity in the train rows; it sums to 1, favours features with many distinct values, and is computed on data the model has already seen. The permutation importance shuffles one column of the TEST rows and measures how much the score falls: a feature that matters hurts when scrambled, a useless one does not. On an artificial problem whose label depends on the product of two features, a logistic regression has an AUC near 0.5 while trees find the interaction, and both rankings point to the right two columns. On SPY direction the picture is the opposite of useful: the unlimited tree scores 50.5% on test, the AUCs are 0.498 (logit), 0.515 (forest) and 0.531 (boosting), the six impurity importances are all between 0.13 and 0.20, and the permutation ranking disagrees with them. Flat importances and a test AUC near 0.5 say that there is nothing to rank.
- Limit the depth: an unlimited tree scores 100% on train and about 50% on test.
- Impurity importance is computed on train rows; permutation importance on test rows is the one to trust.
- Trees see interactions a linear model cannot, but noisy returns offer none: expect AUCs near 0.5.
Lab · what's graded
- The depth curve15 pts
- Forest and boosting10 pts
- Three models, three AUCs20 pts
- Impurity importances10 pts
- The permutation ranking20 pts
- The SPY study25 pts