Peeking and stopping early

You start an A/B test planned for two weeks. On day three, version B is ahead with p = 0.03. Stop and ship?

No. Checking results again and again, and stopping as soon as p dips below 0.05, is called peeking. It raises the false-positive rate far above 5%.

Why

The p-value wanders up and down as data arrives. Even with no real difference, it will dip below 0.05 at some point if you look often enough. Stop at the first dip, and you've caught a random wobble.

Johari and colleagues showed that with continuous monitoring, the chance of a false "win" can rise to several times the intended 5%.

What to do

• Fix the sample size or duration in advance, and analyse once at the end.
• If you must monitor, use methods designed for it: sequential tests or "always-valid" p-values, which adjust for repeated looks.
• Separate safety monitoring (stop if something is broken or harmful) from decision-making (which version wins).

Outside A/B tests

The same trap exists everywhere: collecting more participants "until it's significant", or training "until the validation score beats the baseline". Decide the stopping rule first.

RememberDecide when you'll stop before you start. Stopping at the first good-looking p-value is a false positive waiting to happen.