A forecast is only as good as its test
An ARIMA(p,d,q) model explains a series yₜ from its own past and from past shocks, after differencing d times: with d=0, an AR(1) is yₜ=c+φ yₜ₋₁+eₜ, which is mean-reverting when |φ|<1 and has the long-run mean μ=c/(1-φ). SARIMAX(y, order=(p, d, q), trend='c') fits it by maximum likelihood; trend='c' adds the constant, which makes sense only when d=0 (a differenced series has no level to return to). The fitted object gives params, aic and get_forecast(steps). The AIC =2k-2ln L trades fit against the number of parameters k; among candidates fitted to the same data, the lowest AIC is preferred, but only a difference of a few points means anything. For monthly log volatility of SPY, an ARMA(1,1) (AIC 271.9) and an AR(2) (272.7) edge out the AR(1) (275.0), and differencing is far worse (290.5).
In-sample fit says little about forecasting. The honest test is a rolling origin: at each origin t₀, fit on y_1..t₀ only, forecast h steps ahead, and compare with the value that actually came, y_(t₀+h). Collect the errors and score them with the RMSE, √(mean(e²)). Always include baselines: the naive forecast (the last value) and the historical mean. The skill 1-RMSE_model/RMSE_baseline is positive when the model wins. Here, one month ahead the AR(1) has an RMSE of 0.373 against 0.451 (naive) and 0.419 (mean): skill 0.17 and 0.11. Twelve months ahead its RMSE is 0.526, identical to the mean's, and its skill over the naive forecast (0.34) comes only from the naive forecast's failure: the AR(1) forecast has simply decayed to the mean, from -2.09 to -2.01. Knowing how fast a forecast reverts to the mean tells you how far ahead the model knows anything.
- Constant only without differencing; compare candidate orders by AIC and trust only gaps of a few points.
- Rolling origin: refit at each origin on past data only; never score a forecast on data the fit saw.
- Always beat a baseline: the naive forecast and the historical mean; skill 1-RMSEₘ/RMSE_b.
Lab · what's graded
- The SARIMAX fit10 pts
- Point forecasts10 pts
- Comparing orders15 pts
- Rolling-origin errors25 pts
- The RMSE5 pts
- Forecast skill10 pts
- The volatility study25 pts