Reading critically

Reading critically does not mean looking for reasons to dislike a paper. It means checking whether the evidence supports the claims.

Questions to ask every paper

1. What exactly is claimed? Write the main claim in one sentence. Is it about cause, or only association?
2. Does the evidence match the claim? A test on one dataset does not show a method works "in general".
3. Are the comparisons fair? Were the baselines strong, recent and tuned as carefully as the new method? Same data, same compute budget?
4. Is it big enough? How many samples, runs or participants? Is variance reported?
5. What is missing? Ablations that show which part matters? Results on harder data? A failure analysis?
6. Can it be checked? Are code and data shared? Is the method described in enough detail to repeat?
7. Who paid for it? Read the funding and conflict-of-interest statements.

Red flags

• Big words ("revolutionary", "solves") with small evidence.
• Results only on the authors' own new dataset.
• No limitations section, or one that is a single line.
• Only the best examples shown.
• Numbers in the abstract that do not match the tables.

Write a short review

Practise by writing a half-page review of each important paper you read:

• Summary: what the paper does, in your own words (3 sentences).
• Strengths: what is good and convincing.
• Weaknesses: what is unclear, missing or not supported.
• Questions: what you would ask the authors.

This is what peer reviewers do. Doing it often makes you a better reader, and a better writer of your own papers.

RememberBe fair. Every paper has weaknesses. The question is whether the main claim still stands.