Drift detection and change points

When the market changes its mind

A series is stationary if its statistics, such as the mean and the spread, stay the same over time. Markets are not. A strategy may have a steady mean for years and then lose it, volatility may sit near $7\%$ and then jump, and a correlation may break in a crisis. These changes come in three shapes: a sudden jump (a level shift), a slow slide (drift) and a switch between regimes. The SPY return has an annual volatility of $6.8\%$ in 2017, $41.1\%$ in 2008 and $33.7\%$ in 2020. A single estimate from the whole history describes none of these years well, so we need to notice changes and then adapt.

x_t=\frac{r_t^{2}}{\sigma_0^{2}},\qquad S_t=\max\left(0,\ S_{t-1}+x_t-1-k\right),\qquad S_t>h

(1)

The CUSUM (cumulative sum) detector is the classic tool. Fix a baseline volatility $\sigma_0$ from a calm period and measure each day by $x_t=r_t^{2}/\sigma_0^{2}$ , which averages $1$ when nothing has changed. Add $x_t-1-k$ to a running total $S$ each day, where $k$ is an allowance that ignores small excesses, and never let the total go below zero. When the total passes a threshold $h$ , raise the alarm. Resetting at zero is the key: a long calm stretch cannot bank credit that later hides a real change. A small $k$ or $h$ reacts faster but raises more false alarms.

A hand run and real SPYTake

x=0.5,\ 3,\ 4,\ 2,\ 0.2

with

k=0.5

and

h=4

. The total goes

S=0,\ 1.5,\ 4.0,\ 4.5,\ 3.2

. It first passes

4

on day

4

. A

-2\%

day, with

\sigma_0=0.431\%

per day, gives

x=(2/0.431)^{2}=21.5

, a huge step. On SPY, with the baseline taken from 2017 (

6.85\%

per year),

k=0.5

and

h=5

, the first alarm is on 26 January 2018, days before the February 2018 sell-off, and by 16 March 2020 the total has grown to over

3000

. The variance ratio between 2008 and 2017 is

(41.1/6.75)^{2}=37

times.

Detection is only half the job. A particle filter can also raise a flag by itself: when today's return is very unlikely under the current cloud, the average likelihood is tiny and the effective sample size collapses, as on 10 October 2018. After an alarm you can widen the filter's noise so that it follows the new level faster, restart it from a flatter belief or switch to an estimator with a shorter memory, which is the next topic.

Three shapes of change: a jump, a drift and a switch of regime.
CUSUM: a running total of excess evidence that is reset at zero and raises an alarm above a threshold.
Speed against false alarms: a lower threshold finds changes sooner but cries wolf more often.

Lab · what's graded

The total on day two10 pts
The total on day four10 pts
The first alarm day10 pts
A standardised squared return15 pts
A variance ratio10 pts
Lowering the threshold10 pts
A monitoring loop10 pts
Match the vocabulary10 pts
Why reset at zero15 pts

When the market changes its mind

x_t=\frac{r_t^{2}}{\sigma_0^{2}},\qquad S_t=\max\left(0,\ S_{t-1}+x_t-1-k\right),\qquad S_t>h

(1)

A hand run and real SPYTake

x=0.5,\ 3,\ 4,\ 2,\ 0.2

with

k=0.5

and

h=4

. The total goes

S=0,\ 1.5,\ 4.0,\ 4.5,\ 3.2

. It first passes

4

on day

4

. A

-2\%

day, with

\sigma_0=0.431\%

per day, gives

x=(2/0.431)^{2}=21.5

, a huge step. On SPY, with the baseline taken from 2017 (

6.85\%

per year),

k=0.5

and

h=5

, the first alarm is on 26 January 2018, days before the February 2018 sell-off, and by 16 March 2020 the total has grown to over

3000

. The variance ratio between 2008 and 2017 is

(41.1/6.75)^{2}=37

times.

Three shapes of change: a jump, a drift and a switch of regime.
CUSUM: a running total of excess evidence that is reset at zero and raises an alarm above a threshold.
Speed against false alarms: a lower threshold finds changes sooner but cries wolf more often.

Lab · what's graded

The total on day two10 pts
The total on day four10 pts
The first alarm day10 pts
A standardised squared return15 pts
A variance ratio10 pts
Lowering the threshold10 pts
A monitoring loop10 pts
Match the vocabulary10 pts
Why reset at zero15 pts

Drift detection and change points

When the market changes its mind

Lab · what's graded

Log in to run your code.

Drift detection and change points

When the market changes its mind

Lab · what's graded

Log in to run your code.