Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark

Nikita Nangia, Samuel R. Bowman

Abstract: The GLUE benchmark is a suite of language understanding tasks which has seen dramatic progress in the past year, with average performance moving from 70.0 at launch to 83.9. Here, we measure human performance on the benchmark to learn whether significant headroom remains for further progress. We provide a conservative estimate of human performance through crowdsourcing: our annotators are non-experts who must learn each task from a brief set of instructions and 20 examples. In spite of limited training, these annotators robustly outperform the state of the art on six of the nine GLUE tasks and achieve an average score of 87.1. Given the fast pace of progress, the headroom we observe is limited. To reproduce the data-poor setting that our annotators must learn in, we also train the BERT model in limited-data regimes, concluding that low-resource sentence classification remains a challenge for modern neural network approaches.

Introduction

The General Language Understanding Evaluation (GLUE) benchmark was designed to track progress in Natural Language Processing (NLP). However, progress has been unexpectedly rapid. Within a year of its release, state-of-the-art (SOTA) models like BERT and BigBird jumped from an average score of 70.0 to nearly 84.0. This raised an urgent question: Are these benchmarks nearly "solved," or is there still significant "headroom" for models to improve? To answer this, a reliable human baseline is required. This paper establishes a uniform, conservative estimate of human performance across all nine GLUE tasks to determine exactly how close modern models are to human capability.

Methods and Datasets

The study evaluates the full GLUE benchmark, which consists of nine tasks categorized into single-sentence tasks (CoLA, SST-2), similarity/paraphrase tasks (MRPC, STS-B, QQP), and natural language inference (MNLI, QNLI, RTE, WNLI).

Examples of sentence similarity tasks from the GLUE benchmark, illustrating the nuance required for classification.

To ensure the estimate was "conservative," the authors hired non-expert crowdworkers via the Hybrid platform. For each task, workers received brief instructions and 20 practice examples. The final human label was determined by a majority vote of five annotators to reduce individual error. Additionally, to test learning efficiency, the authors trained BERT in "limited-data" regimes using only 500, 1,000, or 5,000 training examples, mimicking the data-poor environment of the human annotators.

Key Results

Performance comparison between humans and models across GLUE tasks, including low-resource BERT variants.

Overall Performance

The human baseline achieved an average score of 87.1, outperforming BigBird (83.9) and BERT (80.5). Humans dominated on tasks requiring linguistic nuance or common sense, such as CoLA (+1.0 over BigBird) and WNLI (+30.8 over BigBird). In WNLI (Winograd Schema NLI), humans scored 95.9 while BERT languished at 65.1, confirming that models still struggle with pronoun reference resolution.

Machine Victories and Data Hunger

Interestingly, models outperformed non-expert humans on tasks with massive training sets, such as QQP (364k examples) and QNLI (108k examples). On QQP, models were over 12 points better than humans. This suggests that given sufficient data, models can identify statistical patterns that humans might find ambiguous or miss during a brief annotation task.

Learning Efficiency

When BERT was restricted to the same small amount of data shown to humans (the "BERT-500" experiment), its performance cratered to an average of 68.5. This highlights a significant gap in sample efficiency; humans are far more capable of learning new linguistic tasks from a handful of examples than current neural architectures.

Discussion and Limitations

The authors acknowledge that 87.1 is likely a lower bound for human performance. Using expert linguists would likely raise scores, particularly on tasks like CoLA. Furthermore, some tasks like QQP were difficult to explain in short instructions, potentially handicapping human performance.

The study concludes that while the performance gap is closing, GLUE still offers headroom for improvement. However, the rapid progress of "Muppet" models (BERT, BigBird, etc.) necessitates the development of more challenging benchmarks, a finding that directly motivated the creation of SuperGLUE.

Summary updated

Abstract

The GLUE benchmark (Wang et al., 2019b) is a suite of language understanding tasks which has seen dramatic progress in the past year, with average performance moving from 70.0 at launch to 83.9, state of the art at the time of writing (May 24, 2019). Here, we measure human performance on the benchmark, in order to learn whether significant headroom remains for further progress. We provide a conservative estimate of human performance on the benchmark through crowdsourcing: Our annotators are non-experts who must learn each task from a brief set of instructions and 20 examples. In spite of limited training, these annotators robustly outperform the state of the art on six of the nine GLUE tasks and achieve an average score of 87.1. Given the fast pace of progress however, the headroom we observe is quite limited. To reproduce the data-poor setting that our annotators must learn in, we also train the BERT model (Devlin et al., 2019) in limited-data regimes, and conclude that low-resource sentence classification remains a challenge for modern neural network approaches to text understanding.
BibTeX
@inproceedings{nangia2019human,
  author = {Nikita Nangia and Samuel R. Bowman},
  title = {{Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=tDmrjR44EUF},
}
Figure