Letting the correlations draw the groups
Hierarchical clustering starts from a distance between every pair of assets. A correlation is not a distance, but dᵢⱼ=√(1/2(1-ρᵢⱼ)) is: it is 0 for ρ=1, about 0.71 for ρ=0 and 1 for ρ=-1, and it obeys the triangle inequality. squareform(d, checks=False) turns the square matrix into the condensed list of n(n-1)/2 distances that scipy.cluster.hierarchy.linkage(y, method='average') expects. The result Z is a tree written as n-1 rows [left, right, height, size]: at each step the two closest groups are merged, and the distance between two groups is the average of the distances between their members.
Two functions read the tree. hierarchy.leaves_list(Z) gives the order of the leaves in which similar assets are neighbours, which reorders a correlation matrix into visible blocks. hierarchy.fcluster(Z, k, criterion='maxclust') cuts the tree into at most k groups and returns a label for each asset. On twelve stocks and ETFs over 1,500 days, with no sector information at all, five groups come out: tech (AAPL, MSFT, NVDA), banks (JPM, BAC, GS), energy (XOM, CVX), health (JNJ, PFE) and bonds (TLT, IEF). hierarchy.cophenet(Z, y) measures how faithful the tree is: it correlates the original distances with the distances the tree implies, and here it gives 0.96. A value near 1 means the tree is a good summary; a low one means the structure is not really a hierarchy.
- Correlation is not a distance: use √(1/2(1-ρ)), then
squareform(..., checks=False). leaves_listorders,fclustercuts: one gives neighbours, the other gives groups.- Cophenetic correlation near 1 means the tree keeps the distances; check it before trusting the groups.
Lab · what's graded
- Correlation distance15 pts
- The average-linkage tree15 pts
- The leaf order15 pts
- Cutting the tree20 pts
- The cophenetic correlation15 pts
- The twelve-asset study20 pts