Tracking something you cannot see
A state-space model separates what is real from what is measured. A hidden state μt, such as the true volatility level, moves on its own, and what you observe is that state plus noise. In the local-level model yt=μt+εt with εt∼N(0,r) and μt=μt−1+ηt with ηt∼N(0,q). The measurement variance r says how noisy a reading is and the level variance q says how fast the truth drifts. In statsmodels the class UnobservedComponents(y, 'llevel') fits this model, and it estimates q and r by maximum likelihood, so you do not have to guess them.
The Kalman filter turns each new reading into an updated belief, in two steps. Predict: the mean stays and the uncertainty grows, P−=P+q. Update: the gain K=P−/(P−+r) says how much of the surprise y−a to believe, the new mean is a+K(y−a) and the new uncertainty is (1−K)P−. When a reading is missing you do only the predict step. The filter uses data up to today, while the smoother looks back from the whole sample to give the best estimate of every past state.
P−=P+q,K=P−+rP−,a′=a+K(y−a),P′=(1−K)P−(1) Three months by handStart with belief
a=10,
P=4 and take
q=1,
r=5. Month one:
P−=5,
K=5/10=0.5, and a reading
12 gives
a=10+0.5⋅2=11 and
P=0.5⋅5=2.5. Month two:
P−=3.5,
K=3.5/8.5=0.4118, a reading
9 gives surprise
−2, so
a=11−0.4118⋅2=10.18 and
P=0.5882⋅3.5=2.06. Month three has no reading: only the predict step runs, so
a stays
10.18 and
P grows to
2.06+1=3.06. On real data, take the log of SPY's annualised realised volatility for each of
259 months since February 2005. Maximum likelihood gives
r^=0.0889 and
q^=0.0449, so
q/r=0.50, and the steady-state gain is
K=0.50: each new month moves the volatility estimate halfway to the latest reading. This is exactly an ARIMA(0,1,1) with moving-average coefficient
−0.498=−(1−K) and the same log-likelihood
−151.09. At the end of the sample the filtered level is
12.4% annualised against a last reading of
11.7%, with a standard error of
0.21 in logs, so a 95% band of about
8% to
19%.
Filtered or smoothedUse the filtered estimate for anything you could have known at the time, such as a trading rule or a backtest. The smoothed estimate uses future data and is only for describing the past.
Lab · what's graded
- The predicted variance10 pts
- The Kalman gain15 pts
- The updated mean15 pts
- The updated variance10 pts
- The next month's gain15 pts
- A missing reading10 pts
- The signal-to-noise ratio25 pts
