Walk-forward validation

Test the way you would have traded

A random split into training and test rows, as in ordinary machine learning, breaks on time series: the training set contains days after test days, and neighbouring days are correlated, so the test is partly a copy of the training. Walk-forward validation respects time. Train on a block of the past, test on the block right after it, slide the window forward and repeat. Every prediction is made by a model that had never seen the future, which is exactly what live trading is.

import numpy as np

n, train_size, test_size = 10, 4, 2
start = 0
while start + train_size + test_size <= n:
    train = np.arange(start, start + train_size)
    test = np.arange(start + train_size, start + train_size + test_size)
    print("train", train.tolist(), "test", test.tolist())
    start += test_size                                   # slide forward by one test block

Rolling or expanding, and the embargo

Two flavours. A rolling window keeps the training length fixed, dropping the oldest rows: it adapts to regime changes and uses constant cost. An expanding window always starts at row 0, so the training set grows: it uses every observation, but old data may be stale. A third ingredient is the gap (or embargo): if a feature or the target looks h days back or ahead, rows just before the test period share information with it, so skip a few rows between train and test.

def walk_forward_splits(n, train_size, test_size, gap=0):
    splits, start = [], 0
    while start + train_size + gap + test_size <= n:
        train = list(range(start, start + train_size))
        test = list(range(start + train_size + gap, start + train_size + gap + test_size))
        splits.append((train, test))
        start += test_size
    return splits

for train, test in walk_forward_splits(12, 5, 2, gap=1):
    print(train, "->", test)                         # the row between them is skipped

Scoring each fold

For each fold fit on the training rows and score on the test rows, so you get one error per fold instead of one number: their spread tells how stable the model is through time. A useful baseline is the trivial forecast, here the training mean for every test day. Plot or print the errors fold by fold: a model that is good on average but terrible in one regime is not the same as a consistently decent one.

import numpy as np

def walk_forward_splits(n, train_size, test_size):
    out, start = [], 0
    while start + train_size + test_size <= n:
        out.append((np.arange(start, start + train_size), np.arange(start + train_size, start + train_size + test_size)))
        start += test_size
    return out

rng = np.random.default_rng(2)
returns = rng.normal(0.0004, 0.01, size=300)
errors = []
for train, test in walk_forward_splits(len(returns), 100, 50):
    errors.append(np.mean((returns[test] - returns[train].mean()) ** 2))
print(np.round(np.array(errors) * 1e4, 3))                      # squared error in basis points squared, fold by fold

Catching a leak

A split is clean only if every training row comes strictly before every test row. Checking train.max() < test.min() takes one line and catches the most common bug: an unsorted or shuffled index, a slice off by one, a split taken from a library function that shuffles by default. Put such a check inside the code that builds the folds, so a leak stops the run instead of producing a flattering result.

import numpy as np

def leaks_future(train_idx, test_idx):
    return bool(np.max(train_idx) >= np.min(test_idx))

train = np.arange(0, 100)
print(leaks_future(train, np.arange(100, 120)))                     # False: clean
shuffled = np.random.default_rng(1).permutation(120)                # what a shuffling splitter does
print(leaks_future(shuffled[:100], shuffled[100:]))                 # True: training rows after test rows

• Walk-forward: train on a block of the past, test on the next block, slide forward; every test row comes after all its training rows.
• Rolling windows keep the training size fixed; expanding windows start at row 0; a gap (embargo) separates train and test when features look over several days.
• Score each fold separately and look at the spread; check train.max() < test.min(); never tune on the folds you report.

Next up: the limit order book, the data structure where prices are actually made.

Lab · what's graded

• Slide a window through time with walk_forward_splits25 pts
• Keep all the history with expanding_splits25 pts
• Test a forecast fold by fold with forecast_mse25 pts
• Catch a leak with leaks_future25 pts