Statistics for Researchers › Module 5: Power and sample size
Planning sample size
Lesson 2 of 3 · 2 min
How many participants, images or runs do you need? You decide this before the study, with a power analysis.
The four linked numbers
| Number | Typical choice |
|---|---|
| Significance level (α) | 0.05 |
| Power | 80% (or 90%) |
| Effect size you want to detect | The smallest effect that would matter |
| Sample size (n) | What you solve for |
- Significance level (α)
- Typical choice: 0.05
- Power
- Typical choice: 80% (or 90%)
- Effect size you want to detect
- Typical choice: The smallest effect that would matter
- Sample size (n)
- Typical choice: What you solve for
Fix any three, and the fourth follows.
A rule of thumb
To compare two groups with a medium effect (Cohen's d = 0.5) at α = 0.05 and 80% power, you need about 64 people per group. For a small effect (d = 0.2), you need about 400 per group. Small effects need big studies.
Choosing the effect size
The hardest part is choosing the effect size. Don't just use the big effect from a small earlier study: it is probably inflated. Ask instead: what is the smallest effect that would be worth knowing about?
Tools
Free tools such as GPower, or functions in R and Python (statsmodels), do the calculation. In ML, the same thinking applies to how many test examples* and how many random seeds you need to tell two models apart.
RememberPlan your sample size before you collect data, based on the smallest effect that would matter.