Bayesian linear regression

Regression with honest uncertainty

Ordinary regression gives you one slope. Bayesian regression gives you a whole distribution for the slope. Take a stock's return $y$ and the market's return $x$ with $y=\beta x+e$, and let the noise $e$ have variance $\sigma^{2}$. Start with a prior for the beta, for example a normal centred on $m_0=1$ with variance $v_0$. The data push the slope toward the least-squares value and the prior pulls it back. It is the same precision-weighted average as in the last topics. The prior precision is $1/v_0$, the data precision is $\sum x^{2}/\sigma^{2}$, and the second one is large when the market moved a lot.

Two things are worth noting. First, a prior centred on $0$ gives exactly ridge regression: the slope is $\sum xy\,/\,(\sum x^{2}+\lambda)$ with $\lambda=\sigma^{2}/v_0$. So ridge regression is just a Bayesian regression whose prior distrusts large slopes. Second, a prediction for a new $x^{*}$ has two sources of spread: the noise $\sigma^{2}$ and the uncertainty of the slope, which adds $x^{*2}/P_n$. The second part grows with the square of $x^{*}$, so predictions far from the data are less certain.

• Bayesian slope: a distribution whose precision is the prior precision plus the data precision.
• Ridge regression is the case of a prior centred on $0$, with $\lambda=\sigma^{2}/v_0$.
• Predictions carry noise plus slope uncertainty, and the second part grows with $x^{*2}$.

Lab · what's graded

• The posterior precision15 pts
• The posterior slope15 pts
• The slope's uncertainty10 pts
• The ridge slope15 pts
• A prediction's spread10 pts
• A very confident prior10 pts
• Match the vocabulary10 pts
• Far from the data15 pts