Baselines and comparisons

"Our method is better" always means better than something. That something is the baseline. A result is only as strong as its baseline.

What a good baseline looks like

• Strong: the best known method, not an old or weak one.
• Fairly tuned: given the same tuning effort and compute as the new method.
• Simple ones too: a basic method that sets a floor. If a big model barely beats a simple one, that matters.
• Same data and metric: otherwise the numbers cannot be compared.

A real warning

In 2018, Melis, Dyer and Blunsom carefully re-tuned standard LSTM language models and found they beat several newer, more complex models that had claimed to be better. Much of the claimed progress came from tuning the new models more than the old ones.

In 2019, Jimmy Lin made a similar point for search and ranking: many papers reported gains over weak baselines, while the gains disappeared against well-tuned older methods.

Questions to ask

1. Is this the strongest baseline available when the paper was written?
2. Were the baseline numbers re-run, or copied from an older paper with different settings?
3. Did both methods get the same tuning budget?
4. Is the gain bigger than the variation between runs?

RememberBetter than what? A strong, fairly tuned baseline makes a result meaningful. A weak one makes it nearly worthless.