The Sensitivity of Language Models and Humans to Winograd Schema Perturbations

Mostafa Abdou, Vinit Ravishankar, Maria Barrett, Yonatan Belinkov, Desmond Elliott, Anders Søgaard

Abstract

Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge (WSC), a widely employed test of common sense reasoning ability. We show, however, with a new diagnostic dataset, that these models are sensitive to linguistic perturbations of the Winograd examples that minimally affect human understanding. Our results highlight interesting differences between humans and language models: language models are more sensitive to number or gender alternations and synonym replacements than humans, and humans are more stable and consistent in their predictions, maintain a much higher absolute performance, and perform better on non-associative instances than associative ones. Overall, humans are correct more often than out-of-the-box models, and the models are sometimes right for the wrong reasons. Finally, we show that fine-tuning on a large, task-specific dataset can offer a solution to these issues.

Introduction

The Winograd Schema Challenge (WSC) was designed as a "Turing Test" for commonsense reasoning. It consists of sentence pairs that differ by only one or two words (the "discriminatory segment"), which flips which person a pronoun refers to (e.g., "The trophy didn't fit into the brown suitcase because it was too [large/small]").

While state-of-the-art models like BERT and RoBERTa have achieved high scores on these benchmarks, researchers suspect a "Clever Hans" effect: are the models actually "thinking," or are they just exploiting statistical quirks in the data? This paper investigates whether these models are truly robust by testing if minor linguistic changes—which don't change the underlying logic for a human—trip up the machines.

Methodology

The authors developed a suite of seven linguistic perturbations applied to the original 285 WSC examples, resulting in a new diagnostic dataset of 2,330 instances. These perturbations were designed to be logically invariant for humans while testing the robustness of language models.

Examples of the seven linguistic perturbations applied to the WSC dataset.

The perturbations include:

  • Tense switch: e.g., changing past to present continuous.
  • Number switch: e.g., changing singular referents to plural.
  • Gender switch: e.g., swapping male names for female names.
  • Voice switch: e.g., changing active voice to passive voice.
  • Relative clause insertion: e.g., adding descriptive clauses.
  • Adverbial qualification: e.g., adding adverbs to the main verb.
  • Synonym/Name substitution: e.g., replacing names with equivalent terms.

Key Results and Analysis

Our evaluation of models like BERT, RoBERTa, and XLNet against human performance revealed significant brittleness in AI reasoning. While humans achieved ~94.4% accuracy and remained highly stable across variations, out-of-the-box models saw significant performance drops.

Change in accuracy (Δ acc) across different perturbations for various models and humans.

Model Brittleness vs. Human Stability

Models are most unstable when faced with word-level changes such as gender, number, or synonym replacements. For example, simply changing a name to a synonym often caused models to flip their answers. Humans, conversely, are slightly more affected by sentence-level complexity (like relative clauses or passive voice), yet they maintain a much higher absolute performance and stability overall.

The Associativity Trap

Models struggle particularly with "non-associative" instances—cases where one cannot rely on simple word-to-word statistical associations (like linking "heavy" and "suitcase"). This suggests that LLMs often rely on statistical shortcuts rather than true commonsense reasoning. Fine-tuning on massive datasets like WinoGrande (40k+ examples) significantly improved accuracy and stability, but it remains an open question whether this constitutes a true advancement in reasoning or just a broader distribution of learned patterns.

Conclusion

While large language models are climbing the leaderboards, they remain surprisingly brittle to minor linguistic variations. Humans remain the gold standard for robust, consistent commonsense reasoning. Future work should explore training models to be perturbation-invariant from the start to move beyond statistical association toward true symbolic understanding.

Summary updated

Abstract

Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of common sense reasoning ability. We show, however, with a new diagnostic dataset, that these models are sensitive to linguistic perturbations of the Winograd examples that minimally affect human understanding. Our results highlight interesting differences between humans and language models: language models are more sensitive to number or gender alternations and synonym replacements than humans, and humans are more stable and consistent in their predictions, maintain a much higher absolute performance, and perform better on non-associative instances than associative ones. Overall, humans are correct more often than out-of-the-box models, and the models are sometimes right for the wrong reasons. Finally, we show that fine-tuning on a large, task-specific dataset can offer a solution to these issues.
BibTeX
@inproceedings{abdou2020sensitivity,
  author = {Mostafa Abdou and Vinit Ravishankar and Maria Barrett and Yonatan Belinkov and Desmond Elliott and Anders Søgaard},
  title = {{The Sensitivity of Language Models and Humans to Winograd Schema Perturbations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=x_7X52fqY41},
}
Figure