Linear models and regularisation

Shrinking coefficients on purpose

Ordinary least squares finds the line that fits the training data best, and with noisy financial data it also fits the noise. Regularisation adds a penalty on the size of the coefficients, so the model gives up a little fit for a lot of stability. Ridge penalises the sum of squared coefficients and shrinks every one of them toward zero. Lasso penalises their absolute values and can push weak coefficients exactly to zero, which selects features. The strength $\alpha$ is a hyperparameter: $\alpha=0$ is plain least squares and a very large $\alpha$ gives a flat model that predicts the average.

In practice, standardise the features inside a pipeline, because a penalty treats every coefficient alike. Choose $\alpha$ on validation data that comes after the training data in time. Start with ridge when features are correlated and the signal is weak, as with momentum, value and quality scores, and use lasso when you want a short list of features. Elastic net mixes both penalties. Neither one creates signal: a penalty can only make a model less sensitive to noise.

• Ridge shrinks, lasso selects: ridge keeps every feature with smaller weights, lasso can drop weak ones.
• Strength is a trade-off: $\alpha=0$ is least squares and a huge $\alpha$ predicts the average.
• Scale the features first: a penalty treats every coefficient alike, so features must be on a common scale.

Lab · what's graded

• Least squares10 pts
• Ridge with alpha 115 pts
• Ridge with alpha 215 pts
• Lasso with alpha 0.515 pts
• Lasso with alpha 110 pts
• A huge penalty10 pts
• Match the vocabulary10 pts
• Why lasso can give exactly zero15 pts