Show Your Work: Improved Reporting of Experimental Results

Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith

Abstract

Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e.g., accuracy) on held-out test data, compared to previous results. In this paper, we demonstrate that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. We argue for reporting additional details, especially performance on validation data obtained during model development. We present a novel technique for doing so: expected validation performance of the best-found model as a function of computation budget (i.e., the number of hyperparameter search trials or the overall training time). Using our approach, we find multiple recent model comparisons where authors would have reached a different conclusion if they had used more (or less) computation. Our approach also allows us to estimate the amount of computation required to obtain a given accuracy; applying it to several recently published results yields massive variation across papers, from hours to weeks. We conclude with a set of best practices for reporting experimental results which allow for robust future comparisons, and provide code to allow researchers to use our technique.

Introduction

In modern Natural Language Processing (NLP), progress is often measured by the ability to achieve the highest accuracy on specific test sets. However, these "state-of-the-art" (SOTA) scores are frequently the result of extensive hyperparameter tuning—running dozens or hundreds of variations of a model to find the single configuration that performs best.

This practice creates a "compute gap": large labs with massive GPU clusters can afford far more trials than smaller teams. If Model A beats Model B, it is often unclear whether Model A is architecturally superior or if the authors simply had the budget to try 10x more hyperparameter settings. Without reporting the computational effort (the "budget") used to reach a result, comparing models fairly or reproducing them reliably becomes nearly impossible.

Methods: Expected Validation Performance

To address this, we propose reporting expected validation performance as a function of the number of hyperparameter search trials ($n$).

Instead of reporting only the single best score found after $N$ total trials, we treat the validation scores from a random search as a distribution. Using a closed-form solution based on cumulative distribution functions (CDFs), we calculate the expected maximum value one would find if they had only run $n$ trials (where $n \le N$). Unlike previous "bootstrap" methods that rely on re-sampling and introduce noise, this method provides an exact calculation based on the observed results of a random search. This allows researchers to plot a curve showing how a model's performance scales with increased compute.

Expected validation accuracy as a function of hyperparameter search budget. The ranking of models (LR vs CNN) can flip depending on the number of assignments allowed.

Key Insights and Results

Our analysis reveals that the "best" model is often budget-dependent. For example, on the Stanford Sentiment Treebank (SST) task, a simple Logistic Regression (LR) model actually outperforms a Convolutional Neural Network (CNN) if the budget is limited to fewer than 10 trials. The CNN only becomes superior once a certain threshold of experimentation is crossed.

Furthermore, some models are "easy to tune" (low variance, high early performance), while others have a "high ceiling" but require massive search budgets to reach their potential. By applying our method to previously published results, we estimated that some SOTA scores likely required massive hidden costs—such as 18 GPU days of tuning for a single model—that were never explicitly reported.

A survey of 50 papers from EMNLP 2018 highlighted the severity of the reporting problem: zero papers reported all the basic experimental details we recommend. While 92% reported data splits, only 14% reported empirical runtime, and only 8% reported hyperparameter search bounds.

Best Practices for Reporting

To improve the reproducibility and fairness of NLP research, we propose a "Best Practices" checklist for experimental reporting. This checklist urges researchers to document infrastructure, runtimes, and hyperparameter search bounds alongside their primary results.

Proposed checklist for reporting experimental results to ensure reproducibility and fair comparison.

Conclusion and Future Work

While our method provides a more robust way to compare models, it has limitations. It currently describes performance within the spent budget ($n \le N$) and cannot yet accurately forecast performance for $n > N$. Additionally, the mathematical model assumes hyperparameters were chosen via random search; while it doesn't directly account for Bayesian Optimization, we still recommend reporting the budget for such methods.

Ultimately, moving beyond single test-set scores toward reporting performance-per-budget curves will allow for a more nuanced and equitable understanding of progress in the field.

Summary updated

Abstract

Research in natural language processing proceeds, in part, by demonstrating that new models achieve superior performance (e.g., accuracy) on held-out test data, compared to previous results. In this paper, we demonstrate that test-set performance scores alone are insufficient for drawing accurate conclusions about which model performs best. We argue for reporting additional details, especially performance on validation data obtained during model development. We present a novel technique for doing so: expected validation performance of the best-found model as a function of computation budget (i.e., the number of hyperparameter search trials or the overall training time). Using our approach, we find multiple recent model comparisons where authors would have reached a different conclusion if they had used more (or less) computation. Our approach also allows us to estimate the amount of computation required to obtain a given accuracy; applying it to several recently published results yields massive variation across papers, from hours to weeks. We conclude with a set of best practices for reporting experimental results which allow for robust future comparisons, and provide code to allow researchers to use our technique.
BibTeX
@inproceedings{dodge2019show,
  author = {Jesse Dodge and Suchin Gururangan and Dallas Card and Roy Schwartz and Noah A. Smith},
  title = {{Show Your Work: Improved Reporting of Experimental Results}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=PS0Zmx8nN-1},
}
Figure