Show Your Work: Improved Reporting of Experimental Results

Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith

Abstract

Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

In modern machine learning, particularly in Natural Language Processing (NLP), the standard for success is often achieving the highest accuracy on a held-out test set. However, these "state-of-the-art" scores are frequently the result of extensive hyperparameter optimization. This paper argues that reporting a single best score is an insufficient and potentially misleading way to compare models. A model that achieves 90% accuracy after 1,000 search trials might be less practical than a model that achieves 89% after only 5 trials.

The authors identify a growing "computation gap" between high-resource labs and the broader research community. Without reporting the budget used to reach a result, it is impossible to determine if a model is architecturally superior or simply better-tuned. This lack of transparency hinders reproducibility and leads to unfair comparisons between models with vastly different tuning budgets.

A New Standard: Expected Validation Performance

To address these issues, the authors propose a new reporting standard: expected validation performance as a function of the computational budget (e.g., the number of hyperparameter trials or training duration).

The Metric and Estimator

Instead of reporting only the single best value found after $N$ trials, researchers should plot a curve showing the expected best value if only $n$ trials had been run (where $n \le N$). The authors derive a closed-form solution for this metric based on the empirical distribution of all validation scores collected during development. Crucially, this method requires no additional computation; it utilizes data already generated during standard hyperparameter searches.

Expected validation accuracy as a function of hyperparameter search budget for Logistic Regression (LR) and a CNN. The "best" model depends on the available budget.

Case Studies and Key Insights

The authors validated their approach across several well-known NLP benchmarks, revealing that model rankings are often budget-dependent.

Budget-Dependent Superiority

On the Stanford Sentiment Treebank (SST) task, Logistic Regression (LR) actually shows higher expected performance than a CNN when the budget is limited to fewer than 10 trials. As shown in Figure 1, the CNN only becomes the "superior" choice once a researcher has the budget to tune it extensively. This demonstrates that the "best" model is not an absolute property but depends on available resources.

Reverse-Engineering "Hidden" Compute

By plotting these curves, researchers can estimate the unreported budgets of previously published results. On the SciTail dataset, the authors compared their expected performance curves for various model families (n-gram, ESIM, DAM, and DGEM) against reported accuracies in the literature.

Comparing reported accuracies to expected performance curves on the SciTail dataset. This visualization reveals the "hidden" computational cost behind published results.

This analysis highlights how current reporting practices obscure the effort required to reach specific performance levels. For example, the authors estimated that the BiDAF model on SQuAD likely required approximately 18 GPU days of tuning to reach its reported score—a detail omitted from the original publication.

The State of Current Reporting

To assess the current landscape, the authors sampled 50 random papers from EMNLP 2018. They found that zero of them reported all the items in their proposed "Results Checklist," which includes details on infrastructure, runtime, and hyperparameter search bounds. Most papers failed to report even basic information like the total number of trials conducted.

Conclusion

The paper advocates for a shift toward more transparent and rigorous reporting. By providing clear visualizations of performance-budget trade-offs, researchers can offer a more complete picture of a model's utility. While the current method assumes random hyperparameter search, it provides a robust foundation for future work. Ultimately, "showing your work" through budget-performance curves is essential for fair comparison and reproducible progress in machine learning.

Summary updated

BibTeX
@inproceedings{dodge2019show,
  author = {Jesse Dodge and Suchin Gururangan and Dallas Card and Roy Schwartz and Noah A. Smith},
  title = {{Show Your Work: Improved Reporting of Experimental Results}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=PS0Zmx8nN-1},
}
Figure