Spatial distances and clustering

Grouping assets by how they move together

A distance says how far apart two things are. For assets, close should mean that they move together, so we turn a correlation $\rho$ into a distance: $d=\sqrt{\tfrac12(1-\rho)}$ . Perfect correlation $\rho=1$ gives distance $0$ , no correlation gives $0.71$ , and perfect opposite movement $\rho=-1$ gives $1$ . Unlike $1-\rho$ , this formula behaves like a real distance, so it can be fed to clustering tools. scipy.spatial.distance.pdist computes all pairwise distances at once, and squareform converts between a full table and the compact list that the clustering functions expect. For finding the nearest neighbours of a point among many, scipy.spatial.cKDTree is much faster than comparing with every point.

Hierarchical clustering starts with every asset alone and repeatedly merges the two closest groups. How to measure the distance between two groups is the linkage rule. single uses the closest pair across the groups, complete uses the farthest pair and average uses the mean of all cross distances. hierarchy.linkage(distances, method) returns the whole merge history, with the height of each merge. hierarchy.fcluster(Z, t, 'distance') then cuts the tree at height $t$ to give group labels, or with 'maxclust' into a chosen number of groups. In finance the groups become a map of diversification: assets in one cluster carry mostly the same risk, so holding five of them is not five bets.

d_{ij}=\sqrt{\tfrac12(1-\rho_{ij})},\qquad d_{\text{average}}(A,B)=\frac{1}{|A||B|}\sum_{i\in A}\sum_{j\in B}d_{ij},\qquad d_{\text{single}}=\min d_{ij},\quad d_{\text{complete}}=\max d_{ij}

(1)

Three assets by handA correlation of

0.5

gives distance

\sqrt{0.25}=0.5

, and

0.8

gives

\sqrt{0.1}=0.316

. Take three assets with distances

AB=0.3

AC=0.8

and

BC=0.7

. The closest pair is

A

and

B

, which merge at height

0.3

. The distance from the new group to

C

\min(0.8,0.7)=0.7

for single linkage,

\max=0.8

for complete linkage and

(0.8+0.7)/2=0.75

for average linkage, so the second merge happens at

0.75

with average linkage. Cutting the tree at height

0.5

leaves two groups,

\{A,B\}

and

\{C\}

. On real data,

55

large US stocks with at least

90\%

of the history give

JPM

and

BAC

a correlation of

0.79

, which is a distance of

0.327

. Average-linkage clustering cut into

6

groups gives one block of

37

stocks, a block of

8

that is exactly the health-care names (JNJ, PFE, MRK, ABT, AMGN, LLY, BMY, MDT), a block of

7

of defensive names (PG, KO, PEP, T, VZ, DUK, SO) and three loners.

Make correlation a distance: $d=\sqrt{(1-\rho)/2}$ is $0$ for twins and $1$ for opposites.
Linkage is a choice: single, complete and average linkage can give different merge heights for the same data.
Clusters map diversification: pick across clusters, because assets within one cluster share most of their risk.

Lab · what's graded

From correlation to distance10 pts
From distance to correlation10 pts
An average-linkage height15 pts
A single-linkage height10 pts
Groups after a cut10 pts
What each routine does15 pts
Clusters and diversification30 pts

Grouping assets by how they move together

d_{ij}=\sqrt{\tfrac12(1-\rho_{ij})},\qquad d_{\text{average}}(A,B)=\frac{1}{|A||B|}\sum_{i\in A}\sum_{j\in B}d_{ij},\qquad d_{\text{single}}=\min d_{ij},\quad d_{\text{complete}}=\max d_{ij}

(1)

Three assets by handA correlation of

0.5

gives distance

\sqrt{0.25}=0.5

, and

0.8

gives

\sqrt{0.1}=0.316

. Take three assets with distances

AB=0.3

AC=0.8

and

BC=0.7

. The closest pair is

A

and

B

, which merge at height

0.3

. The distance from the new group to

C

\min(0.8,0.7)=0.7

for single linkage,

\max=0.8

for complete linkage and

(0.8+0.7)/2=0.75

for average linkage, so the second merge happens at

0.75

with average linkage. Cutting the tree at height

0.5

leaves two groups,

\{A,B\}

and

\{C\}

. On real data,

55

large US stocks with at least

90\%

of the history give

JPM

and

BAC

a correlation of

0.79

, which is a distance of

0.327

. Average-linkage clustering cut into

6

groups gives one block of

37

stocks, a block of

8

that is exactly the health-care names (JNJ, PFE, MRK, ABT, AMGN, LLY, BMY, MDT), a block of

7

of defensive names (PG, KO, PEP, T, VZ, DUK, SO) and three loners.

Make correlation a distance: $d=\sqrt{(1-\rho)/2}$ is $0$ for twins and $1$ for opposites.
Linkage is a choice: single, complete and average linkage can give different merge heights for the same data.
Clusters map diversification: pick across clusters, because assets within one cluster share most of their risk.

Lab · what's graded

From correlation to distance10 pts
From distance to correlation10 pts
An average-linkage height15 pts
A single-linkage height10 pts
Groups after a cut10 pts
What each routine does15 pts
Clusters and diversification30 pts

Spatial distances and clustering

Grouping assets by how they move together

Lab · what's graded

Log in to run your code.

Spatial distances and clustering

Grouping assets by how they move together

Lab · what's graded

Log in to run your code.