What Will it Take to Fix Benchmarking in Natural Language Understanding?

Samuel R. Bowman, George E. Dahl

Abstract

Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.

Introduction: The Paradox of NLU Success

In the field of Natural Language Understanding (NLU), we are currently victims of our own success. Standard benchmarks such as GLUE, SuperGLUE, and SQuAD were designed to measure long-term progress toward human-like linguistic competence. However, state-of-the-art transformer models like BERT and T5 reached "human-level" performance on these leaderboards far faster than the community anticipated.

Despite these high scores, a glaring paradox remains: models that dominate leaderboards frequently fail on simple "stress tests" and exhibit troubling social biases. This suggests that our benchmarks are "saturated"—they no longer provide the headroom necessary to distinguish between a model that has truly mastered a task and one that has simply learned to exploit statistical shortcuts.

The Pitfalls of Adversarial Filtering

The research community has largely reacted to this saturation by turning to adversarial filtering. This involves creating datasets composed specifically of examples that current models fail to solve (e.g., DynaBench). While this makes benchmarks "harder," this paper argues it is a stopgap rather than a solution.

Adversarial filtering is inherently "mode-seeking." It focuses exclusively on what models cannot do today, often at the expense of the broad linguistic foundations they must maintain. By removing examples that models find easy, we may inadvertently remove the very linguistic phenomena we intended to measure. Furthermore, this approach encourages a cycle where researchers build models to avoid the specific "quirks" of the adversary rather than improving general understanding.

A Four-Point Framework for Better Benchmarks

To restore the health of the NLU evaluation ecosystem, we propose that any adequate benchmark must satisfy four fundamental criteria:

  1. Validity: Performance on the benchmark must imply robust performance on the underlying task. It should not be possible for a model to "cheat" by using annotation artifacts or statistical shortcuts that do not generalize.
  2. Reliable Annotation: Labels must be accurate and account for legitimate linguistic ambiguity. We should not penalize models for disagreeing with a single, potentially flawed human label when multiple interpretations are valid.
  3. Statistical Power: As models approach the performance ceiling, datasets must be large enough to distinguish between marginal gains (e.g., 98.0% vs 98.1%) with statistical significance.
  4. Disincentives for Bias: Benchmarks should actively penalize models that rely on harmful social biases—such as gender or racial stereotypes—to make predictions.

Key Insights: Ambiguity and the Bayes Error

A critical oversight in current benchmarking is the assumption that 100% accuracy is the goal. Because human language is inherently ambiguous, the "true" maximum accuracy—the Bayes error—is often much lower than 100%. For example, research indicates that up to 20% of Natural Language Inference (NLI) examples are significantly ambiguous, and in datasets like Natural Questions, up to 36% of examples involve legitimate disagreements between human annotators. When a model "beats" the human baseline on such a dataset, it is likely over-fitting to the specific biases of the annotators rather than resolving the ambiguity correctly.

The Path Forward

Fixing NLU evaluation is a qualitative challenge that requires a significant shift in how we invest in data. We estimate that creating a high-quality, ten-way annotated dataset of 500,000 examples—necessary to maintain statistical power at high accuracy levels—would cost upwards of $1 million.

To meet this challenge, the authors suggest several strategies:

  • Hybrid Data Collection: Combining the scale of crowdsourcing with the precision of expert linguists.
  • Gamification: Using interactive platforms to lower the cost of high-quality data generation.
  • Mandatory Bias Metrics: Making auxiliary bias evaluations (using tools like Winogender) a standard requirement for any model submission.

Ultimately, moving beyond saturated benchmarks requires us to stop chasing higher numbers on flawed datasets and start building evaluation frameworks that reflect the true complexity and social responsibility of human language understanding.

Summary updated

Abstract

Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researchers who develop better systems to demonstrate their improvements. The recent trend to abandon IID benchmarks in favor of adversarially-constructed, out-of-distribution test sets ensures that current models will perform poorly, but ultimately only obscures the abilities that we want our benchmarks to measure. In this position paper, we lay out four criteria that we argue NLU benchmarks should meet. We argue most current benchmarks fail at these criteria, and that adversarial data collection does not meaningfully address the causes of these failures. Instead, restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets, the reliability with which they are annotated, their size, and the ways they handle social bias.
BibTeX
@inproceedings{bowman2021what,
  author = {Samuel R. Bowman and George E. Dahl},
  title = {{What Will it Take to Fix Benchmarking in Natural Language Understanding?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=W3zILB0Yub},
}
Figure