Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark

Nikita Nangia, Samuel R. Bowman

Abstract: The GLUE benchmark is a suite of language understanding tasks which has seen dramatic progress, with average performance moving from 70.0 at launch to 83.9 in early 2019. This paper measures human performance on the benchmark to determine the remaining headroom for progress. Through crowdsourcing, we provide a conservative estimate of human performance: non-expert annotators, trained on only 20 examples per task, achieve an average score of 87.1, outperforming the state of the art on six of nine tasks. However, the gap is narrowing. By training BERT in limited-data regimes, we further demonstrate that low-resource sentence classification remains a significant challenge for modern neural networks compared to human few-shot learning.

Introduction

In 2019, the "BERT revolution" led to rapid improvements on the General Language Understanding Evaluation (GLUE) benchmark. As models like BERT and BigBird (the "Muppets") approached high scores, researchers questioned how much room for improvement remained. Previously, human baselines were often inconsistent, derived from expert linguists or disparate studies. This study establishes a unified, conservative human baseline by testing non-expert crowdworkers under strict constraints, providing a realistic floor for human-level performance that AI should aim to reach.

Methodology

To ensure a conservative estimate, the researchers employed the Hybrid crowdsourcing platform with a specific protocol designed to mimic the evaluation of machine learning models:

  1. Brief Training: Annotators received short instructions and 20 practice examples.
  2. Qualification: Only workers scoring at least 65% on practice sets proceeded.
  3. Annotation: Workers labeled 500 random examples from GLUE test sets.
  4. Majority Vote: Final human labels were determined by the majority vote of five different workers.
  5. Comparison: Human scores were compared against BERT-Large, BigBird, and BERT variants trained on reduced datasets (500 to 5000 examples) to evaluate data efficiency.
Table 1: Performance comparison between humans and models across the GLUE benchmark, including low-data BERT regimes.

Experimental Setup and Datasets

The study evaluates the full GLUE benchmark, which includes:

  • Single-Sentence Tasks: CoLA (linguistic acceptability) and SST-2 (sentiment).
  • Similarity/Paraphrase Tasks: MRPC, STS-B, and QQP.
  • Natural Language Inference (NLI): MNLI, QNLI, RTE, and WNLI.

Of particular interest is WNLI (Winograd NLI), which requires resolving pronoun ambiguity through common-sense reasoning—a task traditionally difficult for neural networks.

Results and Discussion

As shown in Figure 1, the human baseline reached an average of 87.1, surpassing BigBird (83.9) and BERT-Large (80.5).

The Common Sense Gap

The most significant human advantage appeared in WNLI, where humans scored 95.9 compared to the models' ~65.1. This 30-point gap highlights that common-sense reasoning remains a major hurdle for AI. Conversely, models outperformed humans on tasks with massive training sets like QQP (Quora Question Pairs). This suggests that while models can exploit statistical patterns in large datasets, they struggle to match human performance in low-resource settings.

Table 2: Examples from the QQP dataset, illustrating the subtle differences in question pairs that models must distinguish.

Data Efficiency

A key finding is the disparity in data efficiency. Humans achieved their scores after seeing only 20 examples. When BERT was restricted to 500 training examples (as seen in the bottom section of Figure 1), its performance plummeted to an average of 68.5. This confirms that human "few-shot" learning remains vastly superior to current transformer-based architectures.

Ambiguity and Task Difficulty

When analyzing subsets where all five human annotators agreed, the human-model gap widened. This indicates that some human "errors" are actually due to task ambiguity or poor instructions rather than a lack of linguistic understanding. Figure 2 illustrates the types of question pairs found in QQP, where subtle semantic differences can make labeling difficult for non-experts without extensive context.

Conclusion

While the "Muppets" are catching up, humans still hold a significant lead in tasks requiring reasoning and data efficiency. This study's conservative baseline helped catalyze the creation of SuperGLUE, a more challenging benchmark designed to provide more headroom for the next generation of NLP research. Future work must address the challenge of training models that can learn as effectively as humans from minimal data.

Summary updated

Abstract

The GLUE benchmark (Wang et al., 2019b) is a suite of language understanding tasks which has seen dramatic progress in the past year, with average performance moving from 70.0 at launch to 83.9, state of the art at the time of writing (May 24, 2019). Here, we measure human performance on the benchmark, in order to learn whether significant headroom remains for further progress. We provide a conservative estimate of human performance on the benchmark through crowdsourcing: Our annotators are non-experts who must learn each task from a brief set of instructions and 20 examples. In spite of limited training, these annotators robustly outperform the state of the art on six of the nine GLUE tasks and achieve an average score of 87.1. Given the fast pace of progress however, the headroom we observe is quite limited. To reproduce the data-poor setting that our annotators must learn in, we also train the BERT model (Devlin et al., 2019) in limited-data regimes, and conclude that low-resource sentence classification remains a challenge for modern neural network approaches to text understanding.
BibTeX
@inproceedings{nangia2019human,
  author = {Nikita Nangia and Samuel R. Bowman},
  title = {{Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=tDmrjR44EUF},
}

Loading related content...

Figure