Splits and leakage in finance

Skill positionModule 1 of 4 · Topic 3 of 5

1.3 Splits and leakage in finance

—Lab score— / 100Total XP—Streak—

Why the split is the whole game

A model is judged on data it has not seen. In finance there are three sets: training data to fit, validation data to choose settings and test data, used once, to report the result. They must follow time: train on the past, validate on the next stretch, test on the last one. A random split puts days from the same storm in both sets, so the model recognises neighbours instead of learning a rule. Leakage means that information from the validation or test period reaches the training step. It comes in four common forms: a scaler fitted on all days, a feature that uses a future price, a target-like column that is known only afterwards, and labels that overlap in time.

Overlapping labels deserve special care. If the label is the return over the next $H$ months, the label of a training row $i$ and of a test row $j$ share months whenever $i+H>j$ . The last $H-1$ training rows before the test period therefore know part of the test outcome. The cure is purging, which drops those rows, and an embargo, which keeps a safety gap, as the next topics show. A small number shows the size of the problem: with $1000$ monthly rows, a $60/20/20$ split puts the test period at row $800$ , and for $H=3$ two training rows near each boundary overlap.

(i,\,i+H]\cap(j,\,j+H]\neq\varnothing\iff i+H>j\qquad(i<j)

(1)

Numbers and a real exampleIf the training values are

2,4,6,8

the scaler learns mean

5

. Fitted on all days including a test value

20

, it would learn mean

8

, a gap of

3

that came from the future. In real data, take SPY and ask a random forest to predict the next

20

days of volatility from recent volatility. A shuffled 5-fold split gives

R^{2}=0.60

. A time-ordered walk-forward split with a

20

-day purge gives

R^{2}=-2.10

. The shuffled score is higher by

2.70

because neighbouring days share

19

of their

20

label days, and because a time-ordered test has to face storms that it has not seen.

Split by time: train on the past, validate on the next stretch, test on the last, and look at the test once.
Four leaks: fitted preprocessing, future features, target-like columns and overlapping labels.
A shuffled score is a warning sign: if shuffling the days raises the score a lot, the model is recognising neighbours.

Lab · what's graded

Where the test period starts10 pts
The training size10 pts
What the scaler learns from the future10 pts
Overlapping labels15 pts
How much a shuffle flatters10 pts
The right split for a time series10 pts
A leak-free workflow10 pts
Kinds of leakage10 pts
Why shuffling inflates the score15 pts

0/ 9 answered

1Numeric answer10 ptsNot answered

There are $1000$ monthly rows in time order, split $60\%$ training, $20\%$ validation and $20\%$ test. At which row index (counting from $0$ ) does the test period start?

i =

accepted: ± 0.5

2Numeric answer10 ptsNot answered

With the same split of $1000$ rows, how many rows are in the training set?

n =

accepted: ± 0.5

3Numeric answer10 ptsNot answered

The training values are $2,4,6,8$ (mean $5$ ) and the test value is $20$ . A scaler fitted on all five values learns a mean of $8$ . By how much does the mean differ from the honest one?

d =

accepted: ± 0.01

4Numeric answer15 ptsNot answered

The label of row $i$ is the return over the next $H=3$ months. A training row $i$ and the first test row $j=600$ share months when $i+H>j$ . How many training rows $i<600$ overlap the label of row $600$ ?

k =

accepted: ± 0.01

5Numeric answer10 ptsNot answered

On SPY, a random forest predicting the next $20$ days of volatility gets $R^{2}=0.60$ with a shuffled split and $R^{2}=-2.10$ with a purged walk-forward split. By how much does the shuffle flatter it? Give two decimals.

g =

accepted: ± 0.02

6Choice10 ptsNot answered

You want to judge a daily return model honestly. Which split do you use?

7Order the steps10 ptsNot answered

Put the steps of a leak-free evaluation in order.

Drag to reorder, or focus an item, press Space to lift it, the arrow keys to move it and Space to drop it.

1Look at the test score once and report it
2Order the rows by time and cut train, validation and test
3Choose the settings with the validation data
4Fit preprocessing and model on the training data

8Match10 ptsNot answered

Match each kind of leakage with its example.

Pick the partner of each item on the left.

Preprocessing leak

Look-ahead feature

Target leak

Label overlap

9Explain in words15 ptsNot answered

In one or two sentences, explain why a shuffled split gives a better score than a time-ordered split when the target is a return over the next $20$ days.

Graded on:Neighbouring days have almost the same label window, so test days have near copies in the training setSo the model can memorise the neighbours instead of predicting something it has not seen0 / 400 characters

Math modeKeyboard or LaTeX inputTask 1 of 9 · Ctrl ⏎ to check

Splits and leakage in finance

Skill positionModule 1 of 4 · Topic 3 of 5

1.3 Splits and leakage in finance

—Lab score— / 100Total XP—Streak—

Why the split is the whole game

(i,\,i+H]\cap(j,\,j+H]\neq\varnothing\iff i+H>j\qquad(i<j)

(1)

Numbers and a real exampleIf the training values are

2,4,6,8

the scaler learns mean

5

. Fitted on all days including a test value

20

, it would learn mean

8

, a gap of

3

that came from the future. In real data, take SPY and ask a random forest to predict the next

20

days of volatility from recent volatility. A shuffled 5-fold split gives

R^{2}=0.60

. A time-ordered walk-forward split with a

20

-day purge gives

R^{2}=-2.10

. The shuffled score is higher by

2.70

because neighbouring days share

19

of their

20

label days, and because a time-ordered test has to face storms that it has not seen.

Split by time: train on the past, validate on the next stretch, test on the last, and look at the test once.
Four leaks: fitted preprocessing, future features, target-like columns and overlapping labels.
A shuffled score is a warning sign: if shuffling the days raises the score a lot, the model is recognising neighbours.

Lab · what's graded

Where the test period starts10 pts
The training size10 pts
What the scaler learns from the future10 pts
Overlapping labels15 pts
How much a shuffle flatters10 pts
The right split for a time series10 pts
A leak-free workflow10 pts
Kinds of leakage10 pts
Why shuffling inflates the score15 pts

0/ 9 answered

1Numeric answer10 ptsNot answered

There are $1000$ monthly rows in time order, split $60\%$ training, $20\%$ validation and $20\%$ test. At which row index (counting from $0$ ) does the test period start?

i =

accepted: ± 0.5

2Numeric answer10 ptsNot answered

With the same split of $1000$ rows, how many rows are in the training set?

n =

accepted: ± 0.5

3Numeric answer10 ptsNot answered

The training values are $2,4,6,8$ (mean $5$ ) and the test value is $20$ . A scaler fitted on all five values learns a mean of $8$ . By how much does the mean differ from the honest one?

d =

accepted: ± 0.01

4Numeric answer15 ptsNot answered

k =

accepted: ± 0.01

5Numeric answer10 ptsNot answered

g =

accepted: ± 0.02

6Choice10 ptsNot answered

You want to judge a daily return model honestly. Which split do you use?

7Order the steps10 ptsNot answered

Put the steps of a leak-free evaluation in order.

Drag to reorder, or focus an item, press Space to lift it, the arrow keys to move it and Space to drop it.

1Look at the test score once and report it
2Order the rows by time and cut train, validation and test
3Choose the settings with the validation data
4Fit preprocessing and model on the training data

8Match10 ptsNot answered

Match each kind of leakage with its example.

Pick the partner of each item on the left.

Preprocessing leak

Look-ahead feature

Target leak

Label overlap

9Explain in words15 ptsNot answered

In one or two sentences, explain why a shuffled split gives a better score than a time-ordered split when the target is a return over the next $20$ days.

Math modeKeyboard or LaTeX inputTask 1 of 9 · Ctrl ⏎ to check

Splits and leakage in finance

Why the split is the whole game

Lab · what's graded

Log in to run your code.

Splits and leakage in finance

Why the split is the whole game

Lab · what's graded

Log in to run your code.