Rebuilding the method

Keshav's third pass is to virtually re-implement the paper: rebuild the work in your head, or on paper, as if you had to do it yourself. Gaps you would hit become obvious.

The rebuild checklist

For a computational paper, try to write down:

• Data: which dataset, which version, how it was split, what was removed.
• Model: the architecture and its size.
• Training: loss, optimiser, learning rate, batch size, number of steps.
• Tuning: how settings were chosen, and on which data.
• Evaluation: the exact metric, and on which split.
• Compute: the hardware and time used.

Every blank in this list is a place where the result might not be reproducible.

The tuning question

The most important line is often how settings were chosen. If they were tuned on the test set, the reported result is too optimistic. Settings should be chosen on a separate validation set, and the test set used once at the end.

Reproducibility checklists

After the NeurIPS 2019 reproducibility programme, many ML venues began asking authors to complete a checklist: data, code, settings, number of runs, compute, and how error bars were computed. A paper that answers the checklist well is much easier to trust and to rebuild.

RememberTry to write the method out as a recipe. Every blank you cannot fill is a weakness in the paper, or a question for the authors.