Penalties, paths and honest tuning
With sixty noisy regressors, ordinary least squares fits the noise. A penalty adds a price for big coefficients. Ridge(alpha) charges for the sum of squares: every coefficient shrinks, none reaches zero. Lasso(alpha) charges for the sum of absolute values: as alpha grows, coefficients drop to exactly zero one after another, so the lasso also selects variables. ElasticNet(alpha, l1_ratio) mixes the two, which helps when correlated regressors would make the lasso pick one of them at random. The penalty only treats coefficients fairly if the columns are on the same scale, so you standardise first, and with the training mean and deviation only; otherwise the test rows leak into the scale. Looping over a grid of alpha and recording how many coefficients survive, or how large the ridge solution is, gives the regularisation path.
How much penalty? Not the one that fits the past best: that is alpha near zero. You pick alpha by walk-forward testing: for each fold, standardise with the training rows, fit on them, predict the test rows and score the predictions by rank correlation with the outcomes (the information coefficient). If a large alpha zeroes every coefficient the predictions are constant and the correlation is undefined, so score that as 0. For SPY's daily return on the previous-day returns of 60 other assets, the lasso keeps 57 coefficients at α=10⁻⁵, 38 at 10⁻⁴, 15 at 3.2×10⁻⁴ and 7 at 10⁻³, while the ridge norm falls from 0.0043 to 0.0005 without any coefficient becoming zero. The walk-forward IC peaks at α=3.2×10⁻⁴ with about 0.022: a modest, honest number that a penalty-free fit could never have shown you.
- Ridge shrinks, lasso selects, elastic net does both: trace each along a grid of alpha.
- Standardise with the training rows only: a penalty depends on scale, and the test rows must not set it.
- Tune alpha out of sample: score by walk-forward rank correlation, and treat a constant prediction as 0.
Lab · what's graded
- Standardising without peeking10 pts
- The lasso path15 pts
- The ridge path10 pts
- A walk-forward score25 pts
- Choosing alpha15 pts
- What the elastic net keeps10 pts
- The whole path study15 pts