A/B tests

An A/B test is a randomised experiment on a live product: some users see version A, others see version B, and you compare an outcome like clicks, purchases or time saved.

Why companies run thousands

Ron Kohavi, who led experimentation at Microsoft and Amazon, reports that at Microsoft only about one third of ideas tested improved the metrics they were meant to improve. Most ideas, even good-sounding ones, don't work. Without testing, teams would ship them anyway.

In one well-known case at Bing, a small change to how ad headlines were displayed, an idea that had sat untested for months, raised revenue by about 12%. Nobody had predicted it.

Getting it right

• Randomise by user, not by visit, so one person always sees the same version.
• Pick one primary metric in advance, plus "guardrail" metrics that must not get worse (speed, crashes).
• Run for full weeks to cover weekday and weekend behaviour.
• Check the split: if you planned 50/50 and got 52/48, something may be broken (a "sample ratio mismatch").

Beyond companies

The same design works for a study app (two versions of a quiz), a website for your club, or a classroom tool, as long as people are assigned at random.

RememberMost ideas don't work. An A/B test lets real users, assigned by chance, tell you which ones do.