When neighbours share a label
Suppose the label of day t is the return from the close of day t to the close of day t+h, with h=5. Then the labels of day t and day t+1 share four of their five daily returns: they are almost the same number. A K-fold that puts day t in the training set and day t+1 in the test set lets the model see the answer it is being tested on. A flexible model, such as nearest neighbours, then looks skilful just by remembering. The leak has a precise shape: the label of row i is only known at row i+h, so row i overlaps a test block [t₀,t₁] when i+h≥t₀ and i≤t₁.
Purging drops every training row that overlaps the test block. The embargo drops a few more rows just after the block, because slow features such as a 20-day return keep carrying information forward. A splitter for scikit-learn is any object with split(X), which yields (train, test) index arrays, and get_n_splits(); it can then be passed as cv= to cross_val_score or GridSearchCV. On 5,340 days of SPY with four slow features and a five-day label, a nearest-neighbours model has an R squared of 0.039 under shuffled folds and -0.36 under purged folds. The model has no skill, and the shuffled number is the leak speaking. A plain unshuffled K-fold is much better, but it still has 120 overlapping pairs at the four boundaries; the purged splitter has none.
- Overlapping labels put near-copies on both sides of a split; shuffled K-fold then rewards memory.
- Purge rows with i+h≥t₀ and i≤t₁; embargo a few rows after t₁.
- A splitter is just
splitandget_n_splits: any object with both works ascv=.
Lab · what's graded
- Purge and embargo25 pts
- The splitter25 pts
- The scorer10 pts
- Counting overlaps15 pts
- The leakage study25 pts