Abstract
Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e.g., accuracy) on held-out test data, compared to previous results. In this paper, we demonstrate that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. We argue for reporting additional details, especially performance on validation data obtained during model development. We present a novel technique for doing so: expected validation performance of the best-found model as a function of computation budget (i.e., the number of hyperparameter search trials or the overall training time). Using our approach, we find multiple recent model comparisons where authors would have reached a different conclusion if they had used more (or less) computation. Our approach also allows us to estimate the amount of computation required to obtain a given accuracy; applying it to several recently published results yields massive variation across papers, from hours to weeks. We conclude with a set of best practices for reporting experimental results which allow for robust future comparisons, and provide code to allow researchers to use our technique.
Introduction
In modern Natural Language Processing (NLP), progress is often measured by the ability to achieve the highest accuracy on specific test sets. However, these "state-of-the-art" (SOTA) scores are frequently the result of extensive hyperparameter tuning—running dozens or hundreds of variations of a model to find the single configuration that performs best.
This practice creates a "compute gap": large labs with massive GPU clusters can afford far more trials than smaller teams. If Model A beats Model B, it is often unclear whether Model A is architecturally superior or if the authors simply had the budget to try 10x more hyperparameter settings. Without reporting the computational effort (the "budget") used to reach a result, comparing models fairly or reproducing them reliably becomes nearly impossible.
Methods: Expected Validation Performance
To address this, we propose reporting expected validation performance as a function of the number of hyperparameter search trials ($n$).
Instead of reporting only the single best score found after $N$ total trials, we treat the validation scores from a random search as a distribution. Using a closed-form solution based on cumulative distribution functions (CDFs), we calculate the expected maximum value one would find if they had only run $n$ trials (where $n \le N$). Unlike previous "bootstrap" methods that rely on re-sampling and introduce noise, this method provides an exact calculation based on the observed results of a random search. This allows researchers to plot a curve showing how a model's performance scales with increased compute.
Key Insights and Results
Our analysis reveals that the "best" model is often budget-dependent. For example, on the Stanford Sentiment Treebank (SST) task, a simple Logistic Regression (LR) model actually outperforms a Convolutional Neural Network (CNN) if the budget is limited to fewer than 10 trials. The CNN only becomes superior once a certain threshold of experimentation is crossed.
Furthermore, some models are "easy to tune" (low variance, high early performance), while others have a "high ceiling" but require massive search budgets to reach their potential. By applying our method to previously published results, we estimated that some SOTA scores likely required massive hidden costs—such as 18 GPU days of tuning for a single model—that were never explicitly reported.
A survey of 50 papers from EMNLP 2018 highlighted the severity of the reporting problem: zero papers reported all the basic experimental details we recommend. While 92% reported data splits, only 14% reported empirical runtime, and only 8% reported hyperparameter search bounds.
Best Practices for Reporting
To improve the reproducibility and fairness of NLP research, we propose a "Best Practices" checklist for experimental reporting. This checklist urges researchers to document infrastructure, runtimes, and hyperparameter search bounds alongside their primary results.
Conclusion and Future Work
While our method provides a more robust way to compare models, it has limitations. It currently describes performance within the spent budget ($n \le N$) and cannot yet accurately forecast performance for $n > N$. Additionally, the mathematical model assumes hyperparameters were chosen via random search; while it doesn't directly account for Bayesian Optimization, we still recommend reporting the budget for such methods.
Ultimately, moving beyond single test-set scores toward reporting performance-per-budget curves will allow for a more nuanced and equitable understanding of progress in the field.
Summary updated