A sample is not the world

We almost never measure everyone. We study a sample to learn about a population: all the people, images or cases we care about.

Two ways a sample can mislead

• Chance. Even a fair random sample differs a little from the population. Bigger samples differ less. This is what statistics measures.
• Bias. The sample is chosen in a way that differs systematically from the population. Bigger samples do not fix this.

A famous failure

In 1936 the Literary Digest magazine polled about 2.4 million Americans and predicted that Alf Landon would beat Franklin Roosevelt in the US election. Roosevelt won by a landslide. The sample was huge, but it was drawn largely from lists like phone books and car registrations, and only some people replied. It was big and biased.

In research today

• A model trained and tested on photos from one hospital may fail at another.
• A survey shared on social media reaches people who use that platform, and who chose to answer.
• A study of university students may not apply to everyone.

Questions to ask of any sample

1. Who was the population of interest?
2. How was the sample chosen? At random, or by convenience?
3. Who is missing, and could that change the answer?

RememberBigger samples reduce chance error, not bias. Always ask how the sample was chosen and who is missing.