An error bar for every claim
A backtest says a strategy earned 0.05% a day. Is that skill or luck? A hypothesis test answers with a p-value: if the true mean were zero, how often would a sample look this good? stats.ttest_1samp(r, 0.0) tests one mean, and stats.ttest_ind(a, b, equal_var=False) compares two assets without assuming they are equally volatile (Welch's test, the safe default). stats.jarque_bera(r) checks whether returns are normal by looking at skewness and kurtosis. A small p-value is evidence against the idea you tested, not proof of the idea you like. And a test is only as good as its assumptions: the t-test assumes thin tails, which daily returns do not have.
When formulas are doubtful, resample. The bootstrap draws many samples with replacement from your data, recomputes a statistic each time (the mean, a Sharpe ratio, a quantile) and reads an interval from the spread of the results: stats.bootstrap((r,), stat, method='percentile', random_state=np.random.default_rng(seed)). The statistic must accept an axis argument, because SciPy passes a whole batch of resamples at once, and the seed makes the answer repeatable. A permutation test shuffles the labels of two groups to see how often chance alone gives a gap as large as the one observed. Finally, test 62 assets at the 5% level and about three will look significant by luck alone. stats.false_discovery_control(p) applies the Benjamini–Hochberg correction and tells you which discoveries survive.
- Welch by default: do not assume two assets share a volatility.
- Resample when unsure: a bootstrap needs a statistic with an
axisargument and a seed. - Count your tests: 62 tests at 5% give about three false discoveries; correct for them.
Lab · what's graded
- Is the mean zero?8 pts
- Two assets, two volatilities8 pts
- A normality test6 pts
- A bootstrap interval18 pts
- An annualised Sharpe ratio8 pts
- A Sharpe ratio with an error bar10 pts
- A permutation test14 pts
- Counting false discoveries12 pts
- The whole study16 pts