Case: Attention Is All You Need, 2017

The 2017 Transformer paper by Vaswani and colleagues is the basis of most modern language models. Let's apply everything so far.

Claim

A model that uses only attention, with no recurrence and no convolution, gives better translation and trains much faster than the best earlier models.

Evidence

• Translation quality: on the standard WMT 2014 tests, the big Transformer scored 28.4 BLEU on English–German and 41.8 on English–French, beating the best previous results reported, including ensembles.
• Training cost: the big model trained in 3.5 days on 8 GPUs, a small fraction of the cost of earlier top models, which the paper estimates in a table.
• Ablations: a table varies the number of attention heads, the size of the keys, model size and dropout, showing which choices matter.
• Generality: a test on English sentence parsing, a different task, also works well.

Questions a careful reader asks

• Metric: BLEU measures word overlap with reference translations. It is standard, but it is not the same as human judgement of quality.
• Baselines: the comparison uses numbers reported in earlier papers. That is normal, but settings may differ.
• Variance: results across many random seeds are not the focus. How stable are the gains?
• Cost comparison: training cost is estimated, not all measured on the same hardware.

None of these sinks the paper. Later work confirmed its claims again and again. But asking them is exactly what a reviewer does.

RememberEven a landmark paper has limits. Reading it well means seeing both why it convinced people, and what it did not test.