Bayesian thinking in one lesson

Bayesian thinking means starting with what you already believe, then updating it with new evidence. The key lesson: how much evidence should move you depends on how likely the claim was before you saw it.

The medical test puzzle

A disease affects 1 in 1,000 people. A test catches 99% of real cases, and wrongly flags 1% of healthy people. You test positive. How likely is it that you have the disease?

Most people say 99%. Let's count, out of 100,000 people:

• 100 have the disease. The test catches 99 of them.
• 99,900 are healthy. 1% are wrongly flagged: about 999.
• Total positives: 99 + 999 = 1,098. Only 99 are real.

So the chance you have the disease is about 9%, not 99%. The disease was so rare to begin with that most positives are false alarms. This is the base rate.

Why researchers need this

• A surprising result in a field where most new ideas fail is more likely to be a false positive, even at p < 0.05. John Ioannidis made this argument in his 2005 essay "Why most published research findings are false".
• Extraordinary claims need extraordinary evidence. A result that contradicts physics needs far more than one p = 0.04.

Bayesian statistics

Bayesian methods turn this into calculations: a prior belief, combined with the likelihood of the data, gives a posterior belief. They are widely used in ML and increasingly in other sciences. You don't need the maths to use the habit: ask how plausible a claim was before the study.

RememberHow believable a result is depends on how plausible it was before. Rare things stay rare, even after one positive test.