Pilot first

A pilot is a small trial run of your whole experiment before the real thing. Its job is not to answer your question; it's to find out what's broken.

What pilots catch

• Instructions participants misunderstand.
• A survey question with a typo that flips its meaning.
• Code that crashes on the third batch, or a metric computed wrongly.
• A study that takes twice as long as you planned.
• A task so easy (or so hard) that everyone scores the same.

Pilots in ML

Andrej Karpathy's widely shared recipe for training neural networks starts small: look at the data, get a simple baseline working end to end, and overfit a single small batch to prove the pipeline can learn at all, before scaling up.

The rule: don't peek at the answer

Use the pilot to fix procedures, not to choose your hypothesis or analysis. Don't include pilot data in the final results unless you planned to. And don't use the pilot's effect size to plan the sample: tiny pilots give very noisy estimates.

Size

A pilot can be tiny: 5 participants, one dataset, one seed. Its value is in what goes wrong.

RememberRun a small version first, to find what's broken. Fix the procedure, not the hypothesis.