What a p-value is not

p-values are among the most misread numbers in science. Here are the common mistakes.

Five things a p-value is NOT

1. Not the chance the null hypothesis is true. p = 0.03 does not mean "a 3% chance there's no effect".
2. Not the chance your result is a fluke. It is calculated assuming the null is true; it can't tell you how likely that assumption is.
3. Not the size of the effect. A tiny, useless effect can have a tiny p-value with enough data.
4. Not proof of no effect when p is large. p = 0.30 often means "not enough data to tell", not "no difference".
5. Not a measure of importance. Statistical significance is not practical significance.

"Absence of evidence is not evidence of absence"

ExampleA small study of 20 patients finds p = 0.25 for a new drug. The headline says "drug doesn't work". But the 95% CI runs from a small harm to a large benefit. The honest conclusion is: we can't tell yet.

What to say instead

• Instead of "significant" or "not significant", give the estimate, its confidence interval and the p-value itself.
• Say what the interval rules out and what it doesn't.

RememberA p-value is not the chance you're wrong, not the size of the effect, and a large one is not proof of nothing.