Is the average return real?
A backtest shows an average return of +0.14% a day. Is the strategy good, or did it get lucky? Returns are noisy, so any sample has a mean, and even a coin-flip strategy has a positive one half of the time. A hypothesis test asks a precise question: if the true mean were mu0 (usually 0), how surprising would a sample mean this far from mu0 be? The answer is a p-value. A small p-value says the data would be very unlikely under the "nothing is going on" hypothesis.
The t statistic measures the mean in units of its own uncertainty. The uncertainty of a mean is the standard error s / sqrt(n): it shrinks as the sample grows, which is why more data makes the same mean more convincing. Rescaling all returns changes the mean and the standard error alike, so t does not change. As a rule of thumb, |t| above 2 is unlikely to be luck.
import math
import numpy as np
returns = np.array([0.004, -0.002, 0.001, 0.003, -0.004, 0.002, 0.006, -0.001, 0.005, 0.002, 0.003, -0.002])
n = len(returns)
mean = returns.mean()
std_error = returns.std(ddof=1) / math.sqrt(n) # the uncertainty of the mean
t = mean / std_error # testing against mu0 = 0
print(round(mean, 5), round(std_error, 6)) # 0.00142 0.000892 the mean is only 1.6 standard errors from 0
print(round(t, 3)) # 1.589From t to a p-value
Under the null hypothesis, t behaves like a standard normal variable when the sample is large, so the two-sided p-value is the probability of landing farther than |t| from 0 in either direction: erfc(|t| / sqrt(2)). With t = 1.59 that is 11%: an outcome this good happens one time in nine by chance, so we cannot claim a real edge. The usual threshold is 5% (|t| = 1.96). For small samples the true reference is Student's t distribution, which has fatter tails, so the normal p-value is a bit too small; with fewer than 30 points use a proper t table.
p_value = math.erfc(abs(t) / math.sqrt(2))
print(round(p_value, 4)) # 0.1121 an outcome this good happens by chance about 1 time in 9
print(round(math.erfc(1.96 / math.sqrt(2)), 4)) # 0.05 |t| = 1.96 is the usual 5% limit
# Testing against another value: is the mean different from 0.2% a day?
t_other = (mean - 0.002) / std_error
print(round(t_other, 3), round(math.erfc(abs(t_other) / math.sqrt(2)), 3)) # -0.654 0.513Confidence intervals and paired samples
A confidence interval says more than a yes/no test: mean -/+ 1.96 * standard error is a range that contains the true mean about 95% of the time. If the interval contains 0 the mean is not significant at 5%. Our interval, from -0.03% to +0.32% a day, spans zero and is wide compared with the mean. To compare two strategies measured over the same days, test their daily differences against 0. Pairing removes the noise the two share (the market's good and bad days), so it detects an edge much more sharply than two separate tests.
low, high = mean - 1.96 * std_error, mean + 1.96 * std_error
print(round(low, 5), round(high, 5)) # -0.00033 0.00316 the interval contains 0: not significant at 5%
# Paired: the same six days for two strategies
a = np.array([0.004, -0.002, 0.001, 0.003, -0.004, 0.002])
b = np.array([0.002, -0.003, 0.000, 0.001, -0.002, 0.0])
d = a - b
t_paired = d.mean() / (d.std(ddof=1) / math.sqrt(len(d)))
print(round(d.mean(), 5), round(t_paired, 3)) # 0.001 1.581 a daily edge of 0.1%, but only 6 days# Same mean and spread, four times the data: t doubles and the p-value collapses
spread = returns.std(ddof=1)
for n in (12, 48):
t_n = mean / (spread / math.sqrt(n))
print(n, round(t_n, 3), round(math.erfc(abs(t_n) / math.sqrt(2)), 4))
# 12 1.589 0.1121
# 48 3.178 0.0015t = (mean - mu0) / (s / sqrt(n)): the mean in units of its standard errors / sqrt(n); more data, bigger |t|.- The two-sided p-value is
erfc(|t| / sqrt(2))for large samples; |t| > 1.96 means p < 5%; the intervalmean -/+ 1.96 * secontaining 0 means not significant. - Compare strategies on the same days with the t statistic of the daily differences; beware of testing many variants.
Next up: linear regression (OLS), fitting a line that explains one return with another.
Lab · what's graded
- Measure the mean in units of its noise with
t_statistic25 pts - Turn t into a p-value with
two_sided_p25 pts - Bracket the true mean with
confidence_interval25 pts - Compare two strategies day by day with
paired_t_statistic25 pts