Error bars and what they hide

Lesson 2 of 3 · 2 min

Error bars show uncertainty. But there are three common kinds, and they mean very different things. A figure that does not say which one it uses is incomplete.

The three kinds

Bar	Shows	Gets smaller with more data?
SD (standard deviation)	How spread out the individual results are	No
SE (standard error)	How precisely the average is known	Yes
95% CI (confidence interval)	A range likely to contain the true average	Yes

SD (standard deviation): Shows: How spread out the individual results are; Gets smaller with more data?: No

SE (standard error): Shows: How precisely the average is known; Gets smaller with more data?: Yes

95% CI (confidence interval): Shows: A range likely to contain the true average; Gets smaller with more data?: Yes

SE bars are about half the width of 95% CI bars for reasonably large samples. So the same data can look "clearly different" with SE bars and "overlapping" with CI bars.

Rules of thumb

From Cumming, Fidler and Vaux (2007), for two independent groups of reasonable size:

If 95% CI bars overlap by less than about half their arm length, the difference is usually statistically significant.
If SE bars overlap at all, the difference is usually not significant.

These are rough guides, not tests, but they stop the most common misreadings.

What error bars hide

How many runs or people. Bars from 3 runs mean much less than bars from 30.
Repeated measures. If the same people or seeds are used in both groups, the bars can be misleading. The paper should say so.
No bars at all. In ML, many results are a single run. Different random seeds can change results by more than the claimed gain.

RememberAlways find out what the error bars are, and from how many runs. No bars, or unnamed bars, means the uncertainty is unknown.