Information coefficient hit rate and calibration

Scoring a model that ranks stocks

A return model for many stocks does not need to predict the size of each return. It needs to put the stocks in the right order. The information coefficient, IC, is the rank correlation between the predictions and the realised returns of one month, with $\rho=1-6\sum d_i^{2}/(n(n^{2}-1))$ where $d_i$ is the difference between the two ranks of stock $i$. It lies between $-1$ and $1$, and values of $0.02$ to $0.05$ are typical for real signals. You compute one IC per month and look at the mean and at the t-statistic $\overline{\mathrm{IC}}/(s/\sqrt{m})$ over $m$ months, which says whether the mean is distinguishable from zero.

When the model outputs probabilities, such as the chance that a stock beats the median, two more scores matter. The hit rate is the share of right calls, which is easy to read and noisy. The Brier score is the mean of $(p_i-y_i)^{2}$, lower is better, and a constant guess of $0.5$ scores $0.25$. Calibration asks whether the probabilities mean what they say: among the cases where the model said $70\%$, about $70\%$ should have happened. A sorted model can be poorly calibrated, and calibration decides how much to bet. The fundamental law links the two worlds: the annual information ratio is roughly the IC times the square root of the number of independent bets.

• Rank, not size: the IC rewards putting stocks in the right order and ignores outliers' size.
• Look at the t-statistic: a mean IC of $0.03$ is only meaningful if it is several standard errors above zero.
• Probabilities need calibration: Brier score and reliability tell whether $70\%$ means $70\%$.

Lab · what's graded

• Squared rank differences10 pts
• The rank correlation15 pts
• A t-statistic for the mean IC15 pts
• A Brier score15 pts
• The fundamental law10 pts
• Hit rate against IC10 pts
• Match the vocabulary10 pts
• Why calibration matters15 pts