Seeds and variance

Training the same model twice with different random seeds (different initial weights, data order, dropout) gives different results. Sometimes very different.

How big can it be?

In 2018, Henderson and colleagues studied deep reinforcement learning. They ran the same algorithm with the same settings on 10 seeds, split them into two groups of 5, and plotted the averages. The two groups looked like two different algorithms, with a statistically significant gap between them. Only the seeds differed.

What this means

• A single run can't tell you whether method A beats method B.
• Reporting your best seed is a form of cherry-picking.
• Differences smaller than the seed-to-seed spread are not evidence.

What to do

1. Run several seeds for every method (5 is a common minimum; more if results are noisy).
2. Report the mean and spread (SD or a confidence interval), not the best run.
3. Use the same seeds for all methods where possible, so comparisons are paired.
4. Remember other sources of variance too: data splits, data order, hardware.

Bouthillier and colleagues (2021) found that variation from data sampling and other sources can be as large as the gains many papers report.

RememberOne run is an anecdote. Report the mean and spread over several seeds, and compare methods on the same ones.