What Will it Take to Fix Benchmarking in Natural Language Understanding?

Samuel R. Bowman, George E. Dahl

Samuel R. Bowman, George Dahl Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

Abstract

Natural Language Understanding (NLU) benchmarks are saturating, with models "beating" humans on paper while failing simple real-world tests. This position paper argues that current evaluation is broken and proposes four essential criteria—validity, reliability, statistical power, and bias resistance—to build benchmarks that actually measure progress toward true machine intelligence.

Introduction: The Leaderboard Arms Race

In recent years, the NLU community has entered a "leaderboard arms race." Benchmarks like GLUE and SuperGLUE, originally designed to be multi-year challenges, were "solved" by models like BERT and T5 in record time. On paper, these models exhibit "superhuman" performance. However, a glaring gap remains: these same models often fail spectacularly when faced with simple logic, slight variations in phrasing, or out-of-distribution data.

This paper argues that the current benchmarking ecosystem is failing because it rewards models that exploit statistical shortcuts—dataset artifacts—rather than models that possess a genuine understanding of language. As we move forward, the field must shift its focus from raw accuracy to the structural integrity of the benchmarks themselves.

The Four Pillars of Robust Benchmarking

To move beyond the current stagnation, the authors propose a framework built on four essential pillars for future benchmark design:

1. Validity

Performance on a benchmark must correlate with robust performance on the underlying task. Currently, many models achieve high scores by identifying "shortcuts" (e.g., associating specific words with specific labels) rather than performing the intended reasoning. A valid benchmark must be resistant to these artifacts.

2. Reliable Annotation

The field must move beyond simple "majority vote" crowdsourcing. Language is inherently ambiguous—consider the "Is a burrito a sandwich?" problem. Forcing a single "gold" label on ambiguous examples creates noise. The authors suggest that models should perhaps predict a distribution of human labels rather than a single categorical answer.

3. Statistical Power

As models reach 98% or 99% accuracy, the "room" to measure progress shrinks. Detecting a meaningful 1% improvement at this level requires two orders of magnitude more data to maintain statistical significance. Without sufficient statistical power, we cannot distinguish between a genuine breakthrough and a random fluke.

4. Disincentives for Bias

Benchmarks should explicitly penalize models that rely on harmful social biases. Currently, researchers often omit bias metrics (like Winogender) when reporting high leaderboard scores. The authors argue for stricter reporting norms to make these metrics mandatory components of any evaluation.

The "Bayes Error" Mirage

A critical insight of the paper is the "Bayes Error Mirage." We often assume human performance on a dataset is 100%, but due to linguistic ambiguity, the "true" ceiling might actually be closer to 85%. When a model scores 95%, it is likely not "better than humans"; rather, it is overfitting to the specific biases and quirks of the annotators. This suggests that our current metrics for "superhuman" performance are fundamentally flawed.

The Limits of Adversarial Filtering

The authors also critique the trend toward "Adversarial Filtering" (used in platforms like DynaBench). While this method keeps benchmarks "hard" by only keeping examples that fool current models, it can lead to "mode-seeking" behavior. Models may simply learn the specific quirks of the adversary used to filter the data, rather than developing generalizable linguistic skills. This process risks making benchmarks less diverse and harder to validate over time.

The Financial and Cultural Challenge

Fixing these issues is not merely a technical challenge but a financial and cultural one. The authors estimate that a high-quality, ten-way annotated dataset of 500,000 examples would cost over $1 million. Moving away from cheap crowdsourcing toward expert-led or hybrid data collection is slow and expensive.

Furthermore, the research community is currently incentivized to chase raw accuracy on leaderboards. Changing this culture to value robustness and fairness over marginal gains in accuracy will require a shift in how major conferences, such as ACL or NeurIPS, evaluate and highlight research contributions.

Conclusion

If the NLU community does not address these four pillars, we risk a period of stagnation where we continue to build better benchmark-solvers rather than better language-understanders. To achieve true progress, we must invest in the expensive, difficult work of building benchmarks that are as complex and nuanced as the language they seek to measure.

Summary updated

Abstract

Samuel R. Bowman, George Dahl. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
BibTeX
@inproceedings{bowman2021what,
  author = {Samuel R. Bowman and George E. Dahl},
  title = {{What Will it Take to Fix Benchmarking in Natural Language Understanding?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=W3zILB0Yub},
}
Figure