Fair tuning

Most ML methods have settings to tune: learning rate, model size, regularisation. A method given more tuning usually looks better. So an unfair tuning budget can create a "win" that is really just extra effort.

Show your work

In 2019, Dodge and colleagues argued that papers should report how performance changes with the tuning budget: how good each method gets after 1, 10 or 50 tuning trials. Their examples showed that the "best" method can change depending on the budget. A simple model can win at small budgets, and a complex one only at large budgets.

Rules for a fair comparison

• Same budget for every method: the same number of trials, or the same compute.
• Same search method: if you grid-search yours, grid-search the baseline too.
• Same data for tuning: a validation set, never the test set.
• Report the budget and the search ranges, so readers can judge.

A quick check when reading

If a paper tuned its own method carefully but used "default settings" for the baselines, the comparison is tilted. Ask whether the baselines got the same care.

RememberGive every method the same tuning budget, on validation data, and say what that budget was.