Cross-validation and overfitting

Fitting the past is easy

Give a model enough freedom and it will fit any training data, noise included. That is overfitting: a great in-sample fit that collapses on new data. Underfitting is the opposite, a model too simple to capture the pattern. The cure for both is to judge a model on data it has not seen. The in-sample $R^2$ never goes down when you add predictors, so it rewards overfitting, while the adjusted $R^2=1-(1-R^2)\frac{n-1}{n-p-1}$ penalises each extra predictor.

$k$-fold cross-validation splits the data into $k$ folds, trains on $k-1$ of them, scores on the held-out fold and averages the $k$ scores. A separate test set, touched once at the very end, gives the final honest estimate; the validation folds are used up by choices such as the penalty $\lambda$. With time series, shuffling is wrong: a random split trains on days after the ones it tests, which leaks the future. Use walk-forward validation instead, always training on the past and testing on the block that follows, optionally with a gap to avoid overlap in labels.

• Overfitting: great on training data, poor on new data. Underfitting: poor on both.
• Cross-validation: train on some folds, score on the held-out one, average; keep a test set for the end.
• Time series: never shuffle; train on the past, test on the future block (walk-forward).

Lab · what's graded

• The size of a training set10 pts
• Why not shuffle15 pts
• A walk-forward test15 pts
• An adjusted R squared15 pts
• The vocabulary15 pts
• Spot the leakage15 pts
• Why in-sample fit says little15 pts