Groups without labels: regimes and peers
K-means splits n points into k groups by repeating two steps: assign each point to the nearest centre, then move each centre to the mean of its points. It minimises the inertia, the sum of squared distances to the centres, Σᵢ‖xᵢ-c_(g(i))‖². Inertia always falls as k grows, so it cannot choose k alone; the silhouette compares, for each point, the distance to its own group with the distance to the nearest other group, and averages between -1 and 1 (high is well separated). Two practical rules: put the columns on one scale first, because distances add up raw units (StandardScaler), and run several starts (n_init=3) with a fixed random_state, because k-means finds a local optimum and its labels are arbitrary numbers.
For regimes, describe each SPY day by its 21-day return, its annualised 21-day volatility and its drawdown from the running high, standardise, cluster with k=3, and relabel the clusters by volatility so that 0 is the calmest. On 5,384 days the groups are a calm regime on 72% of days (volatility 11.6%), a middle one on 25% (22.4%) and a crisis regime on 2.7% (61.5%). The mean next-day return barely differs (0.04%, 0.05%, 0.14%), but the spread is 0.79%, 1.49% and 3.96%: regimes tell you about risk, not direction. For peers, measure how two assets co-move with the distance 1-ρ, take each asset's row of distances as its description, and cluster those rows. With no sector information, 59 assets fall into tech and consumer, defensives, banks and industrials, energy, and bonds with gold. Check stability before you trust a grouping: the adjusted Rand index of the groups from the first and the second half of the sample is only 0.38, so the borderline assets switch sides.
- Scale first, restart often:
StandardScaler,n_init=3, a fixedrandom_state; labels are arbitrary numbers, so rename them by something meaningful. - Inertia falls with k; the silhouette compares separation to spread, and neither chooses k by itself.
- Regimes tell risk, not direction; check the stability of peer groups with the adjusted Rand index before using them.
Lab · what's graded
- Regime descriptors10 pts
- Fitting regimes20 pts
- Choosing k15 pts
- Summarising regimes15 pts
- Peer groups15 pts
- Stability of the groups10 pts
- The study15 pts