Bias: how evidence fools us

Bias is a systematic error: one that pushes results the same way every time. Unlike random noise, collecting more data does not fix it.

Selection bias

Your sample is not like the group you want to learn about. An online survey about internet use will miss people who are not online. A face dataset collected mostly from one region will not represent faces from everywhere.

Survivorship bias

You only see the cases that "survived". A famous story from the Second World War: the statistician Abraham Wald studied returning bomber planes. Others wanted to add armour where returning planes had the most bullet holes. Wald pointed out that those were the places a plane could be hit and still return. The planes hit elsewhere never came back, so the armour should go where returning planes had no holes. Studying only successful startups or famous researchers has the same problem.

Confirmation bias

We notice and remember evidence that agrees with what we already believe, and explain away the rest. Every researcher has it. The cure is a plan: decide your measure and analysis before you see the results.

Publication bias

Studies with exciting, positive results are more likely to be published than studies that found nothing. So the published record can make an effect look stronger or more common than it really is. This is one reason negative results are worth sharing.

RememberYou cannot remove your own bias by trying harder. You remove it with a method: a plan written in advance, fair comparisons, and other people checking your work.