When two regressors move almost together, such as a stock's momentum over 6 and over 12 months, the regression cannot tell which one deserves the credit. This is multicollinearity. The fit can stay good, but the individual coefficients become unstable: a small change in the data can swing one up and the other down, and their standard errors blow up. The variance inflation factor of regressor j is VIFj=1−Rj21, where Rj2 comes from regressing that regressor on all the others. A VIF above 5 or 10 is a warning.
Regularisation fixes the instability by adding a penalty on the size of the coefficients. Ridge (L2) adds λ∥β∥2: the solution is the ordinary one with λI added to XTX, which makes the matrix well conditioned (tie this to the linear algebra skill) and shrinks every coefficient towards zero. Lasso (L1) adds λ∥β∥1 and can set some coefficients exactly to zero, selecting predictors. Both trade a little bias for a large drop in variance, and λ is chosen by cross-validation, the next topic.
Numbers for correlated predictorsIf regressing predictor j on the others gives Rj2=0.9, then VIFj=1/(1−0.9)=10: its standard error is 10≈3.2 times larger than with uncorrelated predictors. Conversely VIF=5 means Rj2=0.8. With XTX=(4334), ridge with λ=1 uses (5335). In one dimension with XTX=4, XTy=8, OLS gives β^=2 while ridge with λ=4 gives 8/(4+4)=1: the penalty halves the coefficient.
Multicollinearity: unstable coefficients and inflated standard errors, predictions often still fine.
Ridge: shrinks all coefficients, never exactly to zero. Lasso: can set coefficients to exactly zero.
Choose the penalty λ by cross-validation, never by looking at the training fit.
Do not read a coefficient when its twin is in the modelWith two near-duplicate factors, each coefficient alone is not interpretable: a large positive and a large negative one can cancel to give a sensible sum. Judge the pair together, drop or combine one of them, or regularise. Predictions from the model can still be good, because the combination is well determined even when each part is not.
Lab · what's graded
A variance inflation factor15 pts
From VIF back to R squared10 pts
The ridge matrix15 pts
A shrunken coefficient15 pts
Which penalty selects variables15 pts
What collinearity does15 pts
Why ridge helps15 pts
0/ 7 answered
1Numeric answer15 ptsNot answered
Regressing one predictor on all the others gives R squared = 0.9. What is its VIF?
VIF=accepted: ± 0.001
2Numeric answer10 ptsNot answered
A predictor has VIF = 5. What is the R squared of its regression on the other predictors?
R2=accepted: ± 0.001
3Matrix · vector15 ptsNot answered
Let XTX=(4334). Fill in XTX+λI for λ=1.
XTX+I=
4Numeric answer15 ptsNot answered
With one regressor, X^T X = 4 and X^T y = 8. OLS gives 2. What does ridge give with lambda = 4, using X^T y / (X^T X + lambda)?
beta=accepted: ± 0.001
5Choice15 ptsNot answered
Which penalty can set some coefficients exactly to zero?
6Choice15 ptsNot answered
Select every effect of strong multicollinearity.
7Explain in words15 ptsNot answered
In two sentences, say why ridge regression helps when predictors are strongly correlated.
Math modeKeyboard or LaTeX inputTask 1 of 7 · Ctrl ⏎ to check
