When 80% should mean 80%
A probability forecast is calibrated when, among the days where it said p, the share of up days is about p. Ranking and calibration are different virtues: a model can order days well (high AUC) and still say 0.8 when the truth is 0.55. The simplest score of probabilities is the Brier score, 1/nΣ(pᵢ-yᵢ)², the mean squared error of the forecast; 0 is perfect, 0.25 is what you get by always saying 0.5. To see calibration, cut [0,1] into bins, and in each bin compare the mean predicted probability with the share of ones. That table is the reliability curve. Summing the gaps, weighted by the size of each bin, gives the expected calibration error: ECE=Σₖnₖ/n |ȳₖ-p̄ₖ|.
A random forest averages many trees, and averaging pulls probabilities away from 0 and 1 in an odd way: they are often too extreme where the trees agree by memorising noise. CalibratedClassifierCV fixes this by learning a small correction on held-out data. The 'sigmoid' method fits a logistic curve p'=1/(1+e^(-(a s+b))) to the model's scores (two numbers, stable), and 'isotonic' fits a free non-decreasing step function (more flexible, needs more data). With financial data the held-out folds must come later in time than the training folds, so pass cv=TimeSeriesSplit(3). On the last 30% of 5,344 SPY days, always predicting the base rate has a Brier score of 0.2473. The raw forest gets 0.2600 with an error of 0.095, its probabilities running from 0.15 to 0.83; after sigmoid calibration the Brier score is 0.2475 and the error is 0.0001. Calibration turned an overconfident model into an honest one, and the honest one is the base rate, because the forest has no skill.
- Rank is not probability: a high AUC says nothing about whether 80% means 80%.
- Brier score and reliability bins are the checks; the baseline to beat is the Brier score of the base rate.
- Calibrate on later data:
CalibratedClassifierCV(..., cv=TimeSeriesSplit(3)), sigmoid first, isotonic with more data.
Lab · what's graded
- The Brier score10 pts
- The reliability table25 pts
- Calibration error15 pts
- The forest10 pts
- The calibrated forest20 pts
- The calibration study20 pts