Clustering for regimes and peers

Groups without labels

Clustering looks for groups of similar rows without being told what the groups are. K-means is the workhorse. Pick $k$ centres, then repeat two steps: assign each point to its nearest centre, and move each centre to the mean of its points. It stops when nothing changes. The quantity it lowers is the inertia, the sum of squared distances from each point to its centre. Because k-means uses distances, features must be on a common scale, and because it always finds $k$ groups, it will find them even in data that has none. In finance clustering is used to label regimes (calm, stormy, crisis) and to find peers, stocks that behave alike.

\mathrm{inertia}=\sum_{i}\left\|x_i-c_{k(i)}\right\|^{2},\qquad s_i=\frac{b_i-a_i}{\max(a_i,\,b_i)}

(1)

How good is a clustering? The silhouette of a point compares $a$ , its mean distance to the other points of its own cluster, with $b$ , its mean distance to the points of the nearest other cluster: $s=(b-a)/\max(a,b)$ . It is near $1$ for a point deep inside a tight, well separated cluster, near $0$ on a border and negative if the point is probably in the wrong cluster. The mean silhouette helps to compare values of $k$ , but it never proves that clusters are real. Always re-run with other starting points and other features, and ask whether the groups mean something for trading.

Six points and real stocksTake the points

1,2,3,10,11,12

with starting centres

2

and

9

. The points

1,2,3

go to centre

2

and

10,11,12

to centre

9

. The new centres are

2

and

11

. The inertia is

(1+0+1)+(1+0+1)=4

. For the point

3

a=(2+1)/2=1.5

and

b=(7+8+9)/3=8

, so

s=(8-1.5)/8=0.8125

. On the real data,

54

stocks described by their beta to the average stock and their yearly volatility fall into

30

and

24

stocks for

k=2

(silhouette

0.53

), into

25,22,7

for

k=3

(

0.52

) and have a silhouette of

0.46

for

k=4

. For SPY, clustering days by the

20

-day volatility and return finds

72\%

of calm days (volatility

12.4\%

25.7\%

of middling days (

20.3\%

) and

2.3\%

of crisis days (

66.3\%

and a

20

-day return of

-12.7\%

). A hidden Markov model gave different shares, because the features and the method differ.

Two steps: assign to the nearest centre, move the centre to the mean, repeat.
Scale first: distances treat every feature alike, so a large-unit feature would dominate.
Clusters are a lens, not a fact: check stability and meaning before you trade on them.

Lab · what's graded

A new centre10 pts
The inertia10 pts
Distance inside the cluster10 pts
A silhouette15 pts
Days that are not calm5 pts
Why scale the features10 pts
The steps of k-means10 pts
Match the vocabulary10 pts
Why clusters need checking20 pts

Groups without labels

\mathrm{inertia}=\sum_{i}\left\|x_i-c_{k(i)}\right\|^{2},\qquad s_i=\frac{b_i-a_i}{\max(a_i,\,b_i)}

(1)

Six points and real stocksTake the points

1,2,3,10,11,12

with starting centres

2

and

9

. The points

1,2,3

go to centre

2

and

10,11,12

to centre

9

. The new centres are

2

and

11

. The inertia is

(1+0+1)+(1+0+1)=4

. For the point

3

a=(2+1)/2=1.5

and

b=(7+8+9)/3=8

, so

s=(8-1.5)/8=0.8125

. On the real data,

54

stocks described by their beta to the average stock and their yearly volatility fall into

30

and

24

stocks for

k=2

(silhouette

0.53

), into

25,22,7

for

k=3

(

0.52

) and have a silhouette of

0.46

for

k=4

. For SPY, clustering days by the

20

-day volatility and return finds

72\%

of calm days (volatility

12.4\%

25.7\%

of middling days (

20.3\%

) and

2.3\%

of crisis days (

66.3\%

and a

20

-day return of

-12.7\%

). A hidden Markov model gave different shares, because the features and the method differ.

Two steps: assign to the nearest centre, move the centre to the mean, repeat.
Scale first: distances treat every feature alike, so a large-unit feature would dominate.
Clusters are a lens, not a fact: check stability and meaning before you trade on them.

Lab · what's graded

A new centre10 pts
The inertia10 pts
Distance inside the cluster10 pts
A silhouette15 pts
Days that are not calm5 pts
Why scale the features10 pts
The steps of k-means10 pts
Match the vocabulary10 pts
Why clusters need checking20 pts

Clustering for regimes and peers

Groups without labels

Lab · what's graded

Log in to run your code.

Clustering for regimes and peers

Groups without labels

Lab · what's graded

Log in to run your code.