The figure that is missing

Authors choose which figures to show. Naturally, they show the ones that help their claim. A skilled reader asks: what figure would I expect here, and why isn't it shown?

Common missing figures

• Failure cases. Examples where the method does badly.
• Other datasets. A method tested on one dataset may only work on that one.
• Cost. Accuracy plots without training time, memory or compute.
• Variance. Results across several random seeds, not just the best run.
• Simple baselines. A plot that includes the obvious simple method.

How to use this

Before reading the results, sketch the figures you would want if you were trying to test the claim. Then compare with what the paper shows. The difference is your list of questions.

ExampleA paper claims its chatbot is "more helpful". You would want: a comparison with a strong existing chatbot, human ratings from people who do not know which bot is which, and examples of failures. If the paper shows only a few hand-picked good conversations, the most important figures are missing.

Be fair

A missing figure is a question, not proof of a problem. Page limits are real, and some tests are in an appendix or a later paper. Check the appendix and the code before concluding something was hidden.

RememberBefore looking at the results, sketch the figures you would want. What's missing is your list of questions for the authors.