Reviewing ML papers

AI and ML conferences receive tens of thousands of submissions, and many reviewers are students. Venues give guidelines; here is what they consistently ask for.

What ML venues ask reviewers to judge

• Soundness: are the claims supported by the experiments or proofs?
• Presentation: is it clear enough to understand and reproduce?
• Contribution / significance: does it matter to the community?
• Originality: how does it differ from prior work?

NeurIPS, ICLR and others also ask authors to complete a checklist (on code, data, settings, error bars, compute, limitations and broader impacts). Use it: check that the answers are true.

A quick ML review checklist

1. Claim: what exactly is claimed, and in which settings?
2. Baselines: strong, recent, fairly tuned?
3. Variance: multiple seeds, error bars, significance of small gains?
4. Ablations: does each component earn its place?
5. Leakage: could test data have leaked into training or tuning?
6. Reproducibility: code, data, settings, compute reported?
7. Limitations and impact: honestly stated?
8. Related work: are close prior papers missing?

Rebuttals and discussion

Many ML venues let authors respond before the final decision. Read the rebuttal carefully and update your score if your concerns were answered. Engaging in the discussion is part of the job.

Be calibrated

Use the full scale honestly, and state your confidence. A short, specific review is better than a long, vague one.

RememberJudge soundness, clarity, significance and originality, check the authors' checklist, and update your view after the rebuttal.