Showing Your Work Doesn't Always Work

Raphael Tang, Jaejun Lee, Ji Xin, Xinyu Liu, Yaoliang Yu, Jimmy Lin

Abstract

While the NLP community has recently pushed for more rigorous reporting of experimental results—specifically plotting performance against computational budget—this paper reveals that the most popular tool for doing so is mathematically flawed. The authors prove that the "MeanMax" estimator is statistically biased toward underestimation and produces unreliable confidence intervals, potentially leading researchers to draw the wrong conclusions about which models actually perform better.

Introduction

In modern NLP, "explorimentation"—the act of endlessly tinkering with hyperparameters until something works—often masks a lack of scientific rigor. To combat this, a high-profile 2019 paper by Dodge et al. ("Show Your Work") argued that researchers should report the expected validation score as a function of the number of hyperparameter trials (the "budget"). However, Tang et al. argue that if we are going to demand rigor, the tools we use to measure it must be held to the same standard. They identify a critical gap: the "MeanMax" estimator used to calculate these budget-performance curves relies on assumptions that break down in practice, leading to biased results and "brittle" statistical significance.

Methodology

The authors use a "dual-pronged" approach involving mathematical proofs and empirical simulations. First, they compare two types of estimators. The original paper used a $V$-statistic (the MeanMax estimator), which is based on the Empirical Cumulative Distribution Function (ECDF). Tang et al. prove this is inherently biased. They propose an alternative $U$-statistic, which is an unbiased estimator for the expected maximum of $n$ samples.

Second, they employ Monte Carlo simulations using Kernel Density Estimation (KDE) to fit distributions to actual results from common NLP models. This allows for "cleanroom" simulations where the "true" performance is known, enabling them to measure exactly how often the MeanMax estimator gave the wrong answer.

Empirical Analysis

The simulations were based on results from two standard NLP tasks: Document Classification using the Reuters dataset (comparing a Multilayer Perceptron (MLP) and an LSTM) and Sentiment Analysis using the Stanford Sentiment Treebank (SST) (comparing LSTMs with GloVe vs. ELMo embeddings).

Comparison of MeanMax estimates vs. true performance for MLP/LSTM and GloVe/ELMo.

As shown in the performance curves, the MeanMax estimator is "negatively biased," consistently underestimating how good a model actually is for a given budget. This happens because the estimator is built on the ECDF, which struggles to account for the "tails" of a distribution—the very place where the maximum values live.

Key Results

Because the bias isn't uniform across different models, it can lead to "rank reversal." The authors demonstrated cases where the biased estimator showed an MLP outperforming an LSTM, even though the true expected maximum of the LSTM was higher.

Furthermore, the reliability of confidence intervals (CIs) constructed via bootstrapping is severely compromised. The authors proved (Theorem 2) that as the budget $n$ increases, the error in the ECDF grows exponentially. Empirical tests showed that 95% CIs only captured the true value about 70% of the time once the budget reached 20 trials, a massive drop from the nominal coverage researchers expect.

Empirical coverage of 95% confidence intervals vs. number of trials.

Conclusion

The authors derived and provided a $U$-statistic-based estimator that remains unbiased regardless of the budget size, offering a drop-in replacement for more reliable reporting. While they acknowledge that the $U$-statistic might have higher variance in certain scenarios and doesn't account for "wall-clock time," it represents a significant step toward ensuring that when researchers "show their work," the community can actually trust the numbers.

Summary updated

BibTeX
@inproceedings{tang2020showing,
  author = {Raphael Tang and Jaejun Lee and Ji Xin and Xinyu Liu and Yaoliang Yu and Jimmy Lin},
  title = {{Showing Your Work Doesn't Always Work}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.246},
}
Figure