Designing Experiments › Module 4: Controls and checks
Designing an ablation
Lesson 1 of 3 · 2 min
In Deep Paper Reading you learned to read ablations. Here you design one for your own work.
Plan it before you build
List every change your method makes compared with the baseline. Each one is a candidate for removal.
| Version | What it tests |
|---|---|
| Full method | Everything together |
| − component A | Does A matter? |
| − component B | Does B matter? |
| Baseline + A only | Is A enough on its own? |
- Full method
- What it tests: Everything together
- − component A
- What it tests: Does A matter?
- − component B
- What it tests: Does B matter?
- Baseline + A only
- What it tests: Is A enough on its own?
Removing parts ("leave one out") and adding parts ("add one in") answer slightly different questions. Parts can interact: A may only help when B is present.
Make it trustworthy
- Run each version with several seeds, and report the spread.
- Keep everything else fixed: data, budget, tuning.
- Include the ablation for parts that didn't help, too. A component that adds nothing should be removed or reported honestly.
Size it sensibly
If your method has ten parts, you can't test every combination. Prioritise the parts your claim depends on, and the parts that cost the most compute or complexity.
RememberPlan the ablation table before you start. Every part of your method should earn its place.