The Kalman filter recap

The filter for bell curves

When the state moves in a straight-line way and all noise is bell-shaped, something nice happens: the belief about the state stays a bell curve forever. A bell curve needs only two numbers, its mean $m$ and its variance $P$, so a filter has to carry just those two. This is the Kalman filter, and it is the exact answer in this setting. Each day it does two steps. Predict: the mean stays, the variance grows by $q$. Update: the observation $y$ pulls the mean toward itself by a fraction $K$, called the gain, and the variance shrinks.

The update is just a weighted average: the new mean is $(1-K)\,m^{-}+K\,y$. The gain $K$ lies between $0$ and $1$. It is close to $1$ when the prediction is very uncertain (large $P^{-}$) or the observation is precise (small $r$), so you trust the data. It is close to $0$ in the opposite case, so you trust the prediction. The difference $y-m^{-}$ is the surprise, called the innovation. Every update shrinks the variance, because $1-K<1$, while every prediction grows it by $q$. Run long enough and the two balance at a steady level.

The same filter tracks a time-varying beta. The state is the beta $\beta_t$, which drifts like a random walk, and the observation is a stock's return $y_t=\beta_t\,m_t+e_t$, where $m_t$ is the market return. The only change is that the gain becomes $K=P^{-}m_t/(P^{-}m_t^{2}+r)$: a big market move teaches a lot about beta, a day with $m_t\approx0$ teaches almost nothing. On Apple against SPY since 2005, the full-sample beta is $1.07$, while a Kalman filter with $q=10^{-3}$ puts it at $0.77$ at the end of 2008, $1.42$ at the end of 2018 and $0.40$ on the last day, with a standard error of about $0.25$.

• Two numbers per day: a mean and a variance are enough when everything is linear and bell-shaped.
• The gain is a trust dial: $K$ near $1$ trusts the new data, near $0$ trusts the prediction.
• Limits: if the volatility itself is hidden or the noise has fat tails, the belief is no longer a bell curve and we need particles.

Lab · what's graded

• The first gain15 pts
• The updated mean15 pts
• The updated variance10 pts
• Day two15 pts
• The steady level10 pts
• A noisier observation10 pts
• Match the vocabulary10 pts
• Why an update shrinks the variance15 pts