Designing Experiments › Module 6: Planning
Writing a test plan
Lesson 2 of 3 · 2 min
A test plan is a one-page document that fixes every important decision before the experiment runs. It's your pre-registration, your checklist and your methods section draft, all in one.
What goes in it
- Question and hypothesis that could turn out false.
- Design: groups or conditions, the control, randomisation, blinding.
- Primary outcome: exactly one main measure, defined precisely.
- Sample size or number of runs, with a reason.
- Analysis: the comparison or test, and what result counts as support.
- Stopping rule: when data collection ends.
- Exclusions: rules for dropping data, decided now.
- Threats: confounders, leakage risks, and how each is handled.
Make it specific
| Vague | Specific |
|---|---|
| We'll see if the new method is better. | Primary outcome: macro-F1 on the held-out test set, mean of 5 seeds; support if the paired difference's 95% CI is above 0. |
| We'll collect enough data. | 64 participants per group, for 80% power to detect d = 0.5. |
- We'll see if the new method is better.
- Specific: Primary outcome: macro-F1 on the held-out test set, mean of 5 seeds; support if the paired difference's 95% CI is above 0.
- We'll collect enough data.
- Specific: 64 participants per group, for 80% power to detect d = 0.5.
Get feedback
Show the plan to a supervisor or a peer before starting. Problems are cheap to fix on paper and expensive to fix after the data is in.
RememberOne page, written before the data: outcome, sample, analysis, stopping rule. Then follow it.