The Sensitivity of Language Models and Humans to Winograd Schema Perturbations

Mostafa Abdou, Vinit Ravishankar, Maria Barrett, Yonatan Belinkov, Desmond Elliott, Anders Søgaard

Abstract

Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge (WSC), a widely employed test of commonsense reasoning ability. We show, however, with a new diagnostic dataset, that these models are sensitive to linguistic perturbations of the Winograd examples that minimally affect human understanding. Our results highlight interesting differences between humans and language models: language models are more sensitive to number or gender alternations and synonym replacements than humans, and humans are more stable and consistent in their predictions, maintain a much higher absolute performance, and perform better on non-associative instances than associative ones.

Introduction: The "Thinking" Gap

The Winograd Schema Challenge (WSC) was designed as a robust alternative to the Turing Test, focusing on commonsense reasoning. It consists of pairs of sentences that differ by only a "discriminatory segment" (one or two words), which flips the correct referent of a pronoun. For example, in the sentence "The trophy didn't fit into the brown suitcase because it was too large," "it" refers to the trophy; changing "large" to "small" flips the referent to the suitcase.

While recent models like BERT and RoBERTa have achieved scores nearing human performance on the WSC, there is growing concern that these models exploit statistical quirks—such as word co-occurrences—rather than performing genuine reasoning. This study investigates whether language models are truly robust to minor linguistic changes that do not alter the underlying logic of the schema.

Methodology

To test model robustness, the authors developed a diagnostic dataset of 2,330 instances by applying seven linguistic perturbations to the original 285 WSC examples. These perturbations include changes to tense, number, gender, voice, and the addition of relative clauses or adverbs, as detailed in the table below.

Examples of the seven linguistic perturbations applied to the Winograd Schema Challenge.

The study evaluated three "out-of-the-box" models (BERT, XLNet, RoBERTa) and versions fine-tuned on large-scale datasets like WinoGrande. Performance was measured using Accuracy, Stability (consistency of predictions post-perturbation), and Pair Accuracy (correctly solving both halves of a Winograd pair). Human judgments were collected via Amazon Mechanical Turk to establish a baseline for robustness.

Key Results and Insights

The experimental results reveal a significant "stability gap" between humans and language models. Humans are remarkably consistent, with a stability score of approximately 94%, whereas out-of-the-box models are erratic, with stability ranging from 75% to 81%.

Sensitivity to Linguistic Noise

Models often rely on associativity—statistical links between words—rather than reasoning. When perturbations reduce these associations, model performance drops sharply. As shown in the figure below, models exhibit significant negative changes in accuracy (Δ acc) across various perturbations, particularly gender and number switches, where humans remain stable.

Change in accuracy (Δ acc) across different perturbations for various models and humans.

Syntactic Biases and Fine-tuning

We observed that models like BERT possess structural biases, such as a preference for resolving pronouns to the object of a sentence by a margin of 25-30%, regardless of meaning. While fine-tuning on massive datasets like WinoGrande (e.g., RoBERTa+WG) dramatically improves accuracy (87.6%) and stability (95.2%), these models still struggle with complex syntax like passive voice and relative clauses—structures that are underrepresented in training data.

Conclusion

While large-scale fine-tuning can improve benchmark scores, it may simply be "teaching to the test" by learning specific data distributions rather than general reasoning. Humans maintain high accuracy and stability across all perturbations, particularly on non-associative instances. To achieve true human-level commonsense, future AI models must become robust to the "long tail" of linguistic variation. The diagnostic dataset introduced here serves as a vital tool for distinguishing genuine reasoning from sophisticated pattern matching.

Summary updated

Abstract

Large-scale pretrained language models are the major driving force behind recent improvements in perfromance on the Winograd Schema Challenge, a widely employed test of commonsense reasoning ability. We show, however, with a new diagnostic dataset, that these models are sensitive to linguistic perturbations of the Winograd examples that minimally affect human understanding. Our results highlight interesting differences between humans and language models: language models are more sensitive to number or gender alternations and synonym replacements than humans, and humans are more stable and consistent in their predictions, maintain a much higher absolute performance, and perform better on non-associative instances than associative ones.
BibTeX
@inproceedings{abdou2020sensitivity,
  author = {Mostafa Abdou and Vinit Ravishankar and Maria Barrett and Yonatan Belinkov and Desmond Elliott and Anders Søgaard},
  title = {{The Sensitivity of Language Models and Humans to Winograd Schema Perturbations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=x_7X52fqY41},
}
Figure