The quiet leak
Leakage means that information from the test rows reaches the model before it is scored. The loud forms are easy to avoid (a feature built from tomorrow's price). The quiet form hides in preprocessing: you pick the best columns, or scale the data, or fill gaps, using all the rows, and only then split for cross-validation. Every test fold has already influenced the choice, and the score is flattering. Feature selection is the worst offender, because a hundred noisy columns always contain a few that look predictive in any sample; if you choose them on all the data, the test folds are partly 'predicted' by columns that were picked because they fitted those very folds. The structural fix is a Pipeline: when you give cross_val_score the whole pipeline, it refits scaling, selection and the model on each training window and only then scores the test window.
The lab uses a deliberately hard question: can yesterday's returns of 60 other assets predict whether SPY rises today? There is almost nothing to find. The leaky evaluation (SelectKBest(f_classif, k=5) on all rows, then TimeSeriesSplit cross-validation) reports an AUC of 0.534. The honest one (scaler, selector and logistic regression in one pipeline, refitted on each window) reports 0.512. The 0.022 in between is pure leak. You can also look inside: cross_validate(model, X, y, cv=..., return_estimator=True) returns the fitted pipelines, and estimator.named_steps['select'].get_support(indices=True) lists the columns each one chose. Across the five folds only one column is chosen every time, which is what noise selection looks like. A score you computed with a pipeline you can defend; a score computed after 'a little preprocessing on everything' you cannot.
- Split first, then learn anything: scaling, selection and imputation all belong inside the cross-validation loop.
- A Pipeline is a leak guard: give
cross_val_scorethe whole pipeline and every step is refitted on each training window. - Look at what each fold chose: columns that change from fold to fold are noise, not signal.
Lab · what's graded
- The leaky evaluation15 pts
- The pipeline10 pts
- The honest evaluation15 pts
- What each fold chose20 pts
- The size of the leak10 pts
- The real-data comparison30 pts