Effect size: does it matter?

With enough data, almost any difference becomes "statistically significant". The real question is: how big is it? That is the effect size.

Kinds of effect size

• Raw difference: "3 points higher accuracy", "5 minutes faster". Often the clearest.
• Standardised difference (Cohen's d): the difference in means divided by the SD. Useful for comparing studies that use different scales.
• Ratios: risk ratio, odds ratio. Common in medicine.

Jacob Cohen suggested rough labels for d: 0.2 small, 0.5 medium, 0.8 large. He warned they are only a fallback when you have no better sense of what matters in your field.

Significant but tiny

ExampleA study of 1 million users finds that a new button colour raises clicks from 10.00% to 10.05%. p < 0.001. The effect is real, and for a giant website it may even be worth money. For a small app, it's meaningless.

Large but uncertain

A small study can find a big effect with a wide confidence interval. That is a reason to run a bigger study, not to announce a breakthrough.

Always ask

1. How big is the effect, in units people understand?
2. How precise is it (the CI)?
3. Is it big enough to matter for the people who would use it?

RememberSignificance asks "is there an effect?". Effect size asks "does it matter?". The second question is the one people care about.