Deep Paper Reading › Module 4: Methods and results
Ablations: which part did the work
Lesson 3 of 3 · 2 min
New methods usually change several things at once. An ablation study removes one change at a time, to see which ones really matter.
How it works
Start with the full method. Remove part A, measure. Put A back, remove part B, measure. If removing A hurts a lot and removing B changes nothing, then A did the work and B was not needed.
| Version | Error |
|---|---|
| Full method | 10% |
| Without part A | 18% |
| Without part B | 10.2% |
- Full method
- Error: 10%
- Without part A
- Error: 18%
- Without part B
- Error: 10.2%
Here part A matters and part B does not, whatever the paper's story says.
A classic example: ResNet
He and colleagues (2016) found a surprising problem: a plain 34-layer network had higher training error than a plain 18-layer one. Deeper was worse. They then added shortcut connections that let information skip layers. With shortcuts, the 34-layer network beat the 18-layer one.
This comparison is a clean ablation: same depth, same data, with and without the one new idea. It is a big reason the paper was so convincing, and why residual connections are now used almost everywhere.
What to look for
- Does the paper have an ablation table at all?
- Is each new part tested on its own?
- Are the differences bigger than run-to-run noise?
- Does the story in the text match what the table shows?
RememberIf a paper adds three ideas, it should show what each one adds. Without ablations, you do not know which idea to believe.