L1 and L2 regularisation as optimization

Penalties that shrink

Regularisation adds a penalty to a fitting loss so that large coefficients cost something: $\min_{\mathbf w}\ \lVert\mathbf y-X\mathbf w\rVert^2+\lambda\,\Omega(\mathbf w)$. Ridge uses $\Omega=\lVert\mathbf w\rVert_2^2$ and the lasso uses $\Omega=\lVert\mathbf w\rVert_1$. This is an optimization problem like any other, and it has an equivalent constrained form, $\min$ loss subject to $\Omega(\mathbf w)\le t$, in which $\lambda$ is exactly the KKT multiplier of the budget $t$. A larger $\lambda$ means a smaller budget.

The one-variable problems show the difference. Ridge divides the unpenalised value $z$ by $1+\lambda$: every coefficient shrinks by the same proportion but none reaches zero. The lasso subtracts $\lambda$ from $|z|$ and clips at zero, so every coefficient with $|z|\le\lambda$ becomes exactly zero. This is why the lasso selects variables. Geometrically, the $\ell_1$ ball has corners on the axes and the loss contours tend to touch it at a corner, while the round $\ell_2$ ball has no corners. Analytically, $|w|$ has a kink at zero where the slope jumps from $-\lambda$ to $+\lambda$, and a minimum can sit on that kink.

In finance the ridge penalty stabilises regressions on correlated predictors, a cousin of covariance shrinkage, and the lasso builds sparse index-tracking portfolios. One trap: in a long-only, fully invested portfolio $\sum_i|w_i|=\sum_i w_i=1$, so the penalty $\lambda\lVert\mathbf w\rVert_1$ is the constant $\lambda$ and changes nothing. To get few non-zero positions or few trades, penalise the trades $\lVert\mathbf w-\mathbf w'\rVert_1$, which is exactly what turnover and cost limits do in the capstone. In Bayesian terms ridge corresponds to a Gaussian prior on the coefficients and the lasso to a Laplace prior.

• Ridge: $w=z/(1+\lambda)$ in one variable; shrinks smoothly, never exactly to zero.
• Lasso: soft threshold $\mathrm{sign}(z)\max(|z|-\lambda,0)$; sets small coefficients exactly to zero.
• Constrained form: penalty weight $\lambda$ is the KKT multiplier of the budget $\Omega(\mathbf w)\le t$.

Lab · what's graded

• A lasso coefficient10 pts
• A ridge coefficient10 pts
• A coefficient set to zero15 pts
• A one-variable ridge regression15 pts
• Why the lasso makes exact zeros15 pts
• A useless penalty15 pts
• Priors and effects10 pts
• What a larger lambda does10 pts