A hidden level behind noisy measurements
A state space model has a hidden state that evolves in time and noisy observations of it. The simplest, the local level model, is yₜ=μₜ+eₜ with eₜ (variance σ²ₑ), and μₜ=μₜ₋₁+hₜ with hₜ (variance σ²ₕ): the level μₜ is a random walk and each measurement adds independent noise. UnobservedComponents(y, level='llevel').fit(disp=False) estimates σ²ₑ (named sigma2.irregular) and σ²ₕ (sigma2.level) by maximum likelihood. The ratio q=σ²ₕ/σ²ₑ is the signal-to-noise ratio: a small q means a slow level and a lot of noise to average out. For the monthly log volatility of SPY over 258 months, σ̂²ₑ=0.089 and σ̂²ₕ=0.045, so q=0.50.
The Kalman filter estimates the level in two steps each month: predict μ̂_(t|t-1)=μ̂_(t-1|t-1), then correct it with the surprise, μ̂_(t|t)=μ̂_(t|t-1)+Kₜ (yₜ-μ̂_(t|t-1)). After a start-up the gain settles at K=P/(P+1) with P=1/2(q+√(q²+4q)), and the filter is then exactly an exponentially weighted moving average with weight K: K=0.502 here, and pd.Series(y).ewm(alpha=K, adjust=False).mean() matches the filtered level to 7×10⁻¹⁰. The same model is an ARIMA(0,1,1) with moving-average coefficient -(1-K)=-0.498: both have an AIC of 290.526. The filtered level res.filtered_state[0] uses data up to t only. The smoothed level res.smoothed_state[0] uses the whole sample: cut the data at month 200 with the same parameters and the filtered level does not change at all, while the smoothed level moves by 0.116. A smoother is a tool for looking back, never a signal. The forecast of the level is flat at -2.09 and its 95% interval widens from ±0.83 to ±1.61.
- Noise plus level: the signal-to-noise ratio q sets how much each new observation moves the estimate.
- The filter is an EWMA and the local level model is an ARIMA(0,1,1): three names for one idea.
- Filtered uses the past, smoothed uses the future: never trade on a smoothed series.
Lab · what's graded
- The local level fit10 pts
- The two variances10 pts
- Filtered and smoothed levels5 pts
- The steady-state gain15 pts
- The filter as an EWMA15 pts
- Does it look ahead?20 pts
- An ARIMA in disguise10 pts
- The volatility study15 pts