A regime detector you build yourself
In this capstone you build a hidden Markov model from nothing but NumPy and run it on twenty years of SPY. The model says that each day the market is in one of three hidden regimes, calm, normal or stormy, that you never see. Each regime has its own typical return and volatility, and tomorrow's regime depends only on today's through a transition matrix A. The forward pass turns the returns into a chance for each regime on every day. A backward pass adds what the later days say. Viterbi finds the single most likely path. Baum-Welch repeats these steps to learn the regime numbers from the data.
Everything is scaled. Dividing each day's forward numbers by their sum cₜ keeps them from shrinking to zero over thousands of days, and the sum of the log cₜ is exactly the log-likelihood. Viterbi works with logs for the same reason. In the Baum-Welch step, the smoothed chances γ act as weights: each regime's mean and volatility are the weighted mean and weighted spread of the returns, and the transition counts come from the chance of moving from i to j on every day. Each round can only raise the log-likelihood, which is a good test of your code. In the last function you use the fitted regimes to trade: one signal predicts the crisis chance from the day before (honest), the other reads it from the smoothed chances (look-ahead).
- Scale every step: divide by the day's sum and add up the logs, or thousands of days underflow to zero.
- Check with brute force: on a few days you can list every state path and compare. The log-likelihood must never fall from one round to the next.
- Honest signals only: predict tomorrow from today's filtered chances. Smoothed chances flatter the back-test.
Lab · what's graded
- Emission probabilities10 pts
- The forward algorithm20 pts
- The backward algorithm10 pts
- Smoothing5 pts
- Viterbi decoding15 pts
- Baum-Welch fitting25 pts
- The regime study15 pts