The best of many is mostly luck
GridSearchCV tries every setting on a grid, scores each with cross-validation and keeps the best. For financial data the folds must walk forward: TimeSeriesSplit(5) trains on the past and validates on the block that follows, five times. The scoring can be the AUC (scoring='roc_auc'). In a pipeline, a setting is addressed by the step name, two underscores and the parameter: 'logit__C'. After the search, best_params_ holds the winner, cv_results_['mean_test_score'] the score of every grid point, and with the default refit=True the search itself predicts with the best model refitted on all the data it was given. With a handful of settings and a smooth score, the best validation score is a mild overestimate. On our 60 return columns the C grid gives AUCs of 0.512, 0.500, 0.504: tuning changes nothing, because there is nothing to find.
The danger grows with the number of candidates. Draw n models whose true skill is zero; their validation scores scatter around the mean with spread σ, and the best of them sits above the mean by about σ((1-γ)Φ⁻¹(1-1/n)+γΦ⁻¹(1-1/ne)), where γ=0.5772 is Euler's constant and Φ⁻¹ the normal quantile. Try 12 random models that each use three of the 60 columns: the best has a validation AUC of 0.523, and luck alone predicts 0.523. Refit the winner on the training rows and score it on the later 30% of days it never saw: its AUC is 0.478, below a coin. This is the winner's curse. The defences are a final holdout that no search touches, fewer candidates, and a sober look at how much of the best score the formula already explains.
- Walk forward: use
TimeSeriesSplitin the search, never shuffled folds, for time series. - The best of n exceeds the mean by about σ Φ⁻¹(1-1/n) even when nothing works.
- Keep a final holdout that no search touches, and report that score, not the validation one.
Lab · what's graded
- The pipeline5 pts
- The grid search20 pts
- Validation scores10 pts
- Many small models20 pts
- Expected best by luck10 pts
- The winner's curse15 pts
- The study20 pts