Forecast evaluation

Scoring a forecast honestly

A forecast is only worth what it achieves on data it did not see. The honest procedure for time series is walk-forward: fit the model on the past, forecast the next value (or block), record the error, move the window forward and repeat. Never shuffle the observations or split them at random, because a model trained on days after the one it predicts has looked into the future. The forecast error is $e_t=y_t-\hat y_t$. The common scores are the mean absolute error, $\mathrm{MAE}=\frac1n\sum|e_t|$, and the root mean squared error, $\mathrm{RMSE}=\sqrt{\frac1n\sum e_t^{2}}$, which punishes large errors more.

Variance is never observed, so a volatility forecast $h_t$ is scored against a proxy $\tilde\sigma_t^{2}$ such as the squared return or the realised variance. Because the proxy is noisy, the loss must be chosen with care: the squared error on variance is dominated by a few large days, while QLIKE, $L=\tilde\sigma^{2}/h-\ln(\tilde\sigma^{2}/h)-1$, is more robust and charges more for under-predicting variance than for over-predicting it by the same factor. Always compare against a naive benchmark, for example yesterday's value or the historical mean. The out-of-sample $R^{2}_{\mathrm{OOS}}$ compares sums of squared errors: it is positive only when the model beats the benchmark. To decide whether a difference in loss is real or luck, use the Diebold-Mariano statistic: with $d_t$ the loss difference between two models, $\mathrm{DM}=\bar d/\mathrm{se}(\bar d)$ is roughly standard normal, so $|\mathrm{DM}|>1.96$ is significant at 5%. When the forecasts look several steps ahead, the $d_t$ overlap and the standard error must allow for autocorrelation. Choosing the best of many models on the same test data overstates its skill.

• Walk-forward: fit on the past, forecast the next block, move forward; never shuffle time series.
• QLIKE is robust for variance forecasts and punishes under-prediction more; compare with a naive benchmark.
• Diebold-Mariano: $\mathrm{DM}=\bar d/\mathrm{se}(\bar d)$ tests whether two forecasts really differ; $|\mathrm{DM}|>1.96$ is significant.

Lab · what's graded

• Mean absolute error10 pts
• Root mean squared error15 pts
• A QLIKE loss15 pts
• Out-of-sample R squared15 pts
• A Diebold-Mariano statistic10 pts
• Why not shuffle10 pts
• Match the vocabulary15 pts
• Beating a naive benchmark10 pts