Statistics for Researchers › Module 4: Common traps
Multiple comparisons
Lesson 2 of 3 · 2 min
Test one true-null hypothesis at p < 0.05, and you have a 5% chance of a false positive. Test twenty, and you should expect about one false positive, just by chance.
The chance grows fast
| Tests | Chance of at least one false positive |
|---|---|
| 1 | 5% |
| 5 | about 23% |
| 20 | about 64% |
| 100 | over 99% |
- 1
- Chance of at least one false positive: 5%
- 5
- Chance of at least one false positive: about 23%
- 20
- Chance of at least one false positive: about 64%
- 100
- Chance of at least one false positive: over 99%
The dead salmon
In 2009, researchers put a dead Atlantic salmon in a brain scanner and showed it photos of people in social situations. Without correcting for the many thousands of brain regions tested, the analysis found "active" areas in the dead fish's brain. With proper correction, nothing. The study was a deliberate, memorable warning, and it won an Ig Nobel Prize.
Corrections
- Bonferroni: divide 0.05 by the number of tests. Simple and strict.
- False discovery rate (Benjamini–Hochberg): controls the share of "discoveries" that are false. Less strict, common in genomics and ML.
In ML papers
Testing many models, many seeds or many datasets and reporting the best is a multiple-comparison problem too. The best of 20 runs will look better than the method really is.
RememberThe more you test, the more false positives you'll find. Count your tests, and correct for them.