Multiple testing and false discoveries

The more you look, the more you find

A test at the 5% level wrongly flags a real-nothing as significant 5% of the time. Run one test and that is acceptable. Run $m$ independent tests on pure noise and the chance of at least one false positive is $1-(1-\alpha)^m$: with 20 tests it is $1-0.95^{20}\approx64\%$. Trying many signals, parameters, assets or time windows and reporting the one that worked is the most common way quant research fools itself, and it has a name: the multiple-testing problem, or data snooping.

There are two things you can control. The family-wise error rate (FWER) is the probability of making any false discovery; Bonferroni controls it by testing each hypothesis at level $\alpha/m$, which is simple but harsh when $m$ is large. The false discovery rate (FDR) is the expected share of discoveries that are false; the Benjamini-Hochberg procedure sorts the $p$-values, finds the largest $k$ with $p_{(k)}\le kq/m$ and declares the $k$ smallest significant. FDR control is gentler and suits exploratory research where a few false leads are tolerable, FWER control suits the moment you must be sure.

• Many tests: the chance of a false positive grows as $1-(1-\alpha)^m$.
• Bonferroni controls the chance of any false discovery (FWER) with the level $\alpha/m$.
• Benjamini-Hochberg controls the share of false discoveries (FDR) and keeps more power.

Lab · what's graded

• At least one false positive15 pts
• A Bonferroni level10 pts
• Expected false discoveries10 pts
• Benjamini-Hochberg15 pts
• Which controls the false discovery rate15 pts
• The vocabulary15 pts
• The best of many backtests20 pts