Abstract
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
In modern machine learning, particularly in Natural Language Processing (NLP), the standard for success is often achieving the highest accuracy on a held-out test set. However, these "state-of-the-art" scores are frequently the result of extensive hyperparameter optimization. This paper argues that reporting a single best score is an insufficient and potentially misleading way to compare models. A model that achieves 90% accuracy after 1,000 search trials might be less practical than a model that achieves 89% after only 5 trials.
The authors identify a growing "computation gap" between high-resource labs and the broader research community. Without reporting the budget used to reach a result, it is impossible to determine if a model is architecturally superior or simply better-tuned. This lack of transparency hinders reproducibility and leads to unfair comparisons between models with vastly different tuning budgets.
A New Standard: Expected Validation Performance
To address these issues, the authors propose a new reporting standard: expected validation performance as a function of the computational budget (e.g., the number of hyperparameter trials or training duration).
The Metric and Estimator
Instead of reporting only the single best value found after $N$ trials, researchers should plot a curve showing the expected best value if only $n$ trials had been run (where $n \le N$). The authors derive a closed-form solution for this metric based on the empirical distribution of all validation scores collected during development. Crucially, this method requires no additional computation; it utilizes data already generated during standard hyperparameter searches.
Case Studies and Key Insights
The authors validated their approach across several well-known NLP benchmarks, revealing that model rankings are often budget-dependent.
Budget-Dependent Superiority
On the Stanford Sentiment Treebank (SST) task, Logistic Regression (LR) actually shows higher expected performance than a CNN when the budget is limited to fewer than 10 trials. As shown in Figure 1, the CNN only becomes the "superior" choice once a researcher has the budget to tune it extensively. This demonstrates that the "best" model is not an absolute property but depends on available resources.
Reverse-Engineering "Hidden" Compute
By plotting these curves, researchers can estimate the unreported budgets of previously published results. On the SciTail dataset, the authors compared their expected performance curves for various model families (n-gram, ESIM, DAM, and DGEM) against reported accuracies in the literature.
This analysis highlights how current reporting practices obscure the effort required to reach specific performance levels. For example, the authors estimated that the BiDAF model on SQuAD likely required approximately 18 GPU days of tuning to reach its reported score—a detail omitted from the original publication.
The State of Current Reporting
To assess the current landscape, the authors sampled 50 random papers from EMNLP 2018. They found that zero of them reported all the items in their proposed "Results Checklist," which includes details on infrastructure, runtime, and hyperparameter search bounds. Most papers failed to report even basic information like the total number of trials conducted.
Conclusion
The paper advocates for a shift toward more transparent and rigorous reporting. By providing clear visualizations of performance-budget trade-offs, researchers can offer a more complete picture of a model's utility. While the current method assumes random hyperparameter search, it provides a robust foundation for future work. Ultimately, "showing your work" through budget-performance curves is essential for fair comparison and reproducible progress in machine learning.
Summary updated