The Winograd Schemas from Hell

Fábio Gagliardi Cozman, Hugo Munhoz2020 · DOI

TL;DR

As large language models (LLMs) begin to "solve" the Winograd Schema Challenge with 90%+ accuracy, researchers argue they are merely exploiting statistical patterns rather than exhibiting true reasoning. This paper proposes "Extended Winograd Schemas"—diabolically difficult versions involving math, logic, and complex social narratives—to reset the bar for machine intelligence.

What Problem Does This Paper Address?

For years, the Winograd Schema Challenge (WSC) was considered a superior alternative to the Turing Test. A Winograd Schema is a pair of sentences with a pronoun ambiguity that is resolved by a "special word" (e.g., "The trophy doesn't fit in the suitcase because it is too big/small"). Solving these was thought to require "commonsense reasoning."

However, modern LLMs like UnifiedQA have reached human-level performance on standard datasets. The authors argue this is a "false peak." Current schemas are too simple; they rely on obvious causal markers and social norms that are easily captured by the statistical regularities of massive text corpora. In short, machines are "cheating" by using pattern matching instead of actual thinking.

Methods

The authors propose a "working recipe" to build "tough" Winograd Schemas by relaxing the original requirement that they must be so easy for humans that the ambiguity is barely noticed. Their "Extended Winograd Schemas" employ several strategies:

  1. Factual Knowledge: Moving beyond "lowest common denominator" facts to information mastered by an educated undergraduate (e.g., history or literature).
  2. Complex Social Narratives: Using "beautifully crafted" sentences from authors like Jane Austen or Marcel Proust. These require deep social empathy and an understanding of nuanced emotional states rather than simple "A causes B" logic.
  3. Mathematical and Logical Puzzles: Embedding GRE-style math problems or classic logic riddles into the schema format.
  4. Epistemic Reasoning: Creating scenarios where the solution depends on understanding what one character knows about another character's knowledge (e.g., "Cheryl’s Birthday Problem").

Datasets

The paper references the current "gold standards":

  • WSC273: The original small set of 273 schemas.
  • WinoGrande: A much larger, crowdsourced dataset used to train current state-of-the-art solvers.

The authors do not introduce a new large-scale dataset. Instead, they provide paradigmatic examples of "tough" schemas to demonstrate how future datasets should be constructed to resist statistical "clever tricks."

Key Insights and Ideas

  • The "Easiness" Trap: The original requirement that schemas be "easy for humans" is exactly what makes them "easy for computers." If a human can solve it instantly, a statistical model can likely solve it via word co-occurrence.
  • The Undergraduate Union: The authors suggest that "common knowledge" for AI testing should be defined as the union of basic facts known by students across various professions, rather than just "toddler-level" world knowledge.
  • Beyond 50/50: While traditional schemas are binary pairs (50% random guess accuracy), the authors show that complex schemas can involve multiple entities, making random guessing far less effective.
  • Reasoning vs. Recognition: True intelligence requires multi-step reasoning that can be "written down in a page of notes." Current AI fails when the "commonsense" required is removed from simple linguistic patterns.

Key Results

The authors qualitatively tested UnifiedQA (a state-of-the-art solver) on their "diabolical" schemas with telling results:

  • Literary Complexity: On a schema based on Proust, the model provided inconsistent, almost random answers, failing to grasp the narrator's confusion.
  • Logical Puzzles: In a riddle involving four girls and a jewel, the model hallucinated or defaulted to the first name mentioned, failing the logical deduction.
  • Epistemic Reasoning: On "Cheryl’s Birthday" and stamp-color puzzles, the model failed miserably, often providing "trivially correct" but useless answers (e.g., answering "one logician" when asked who had distinct stamps).
  • Sensitivity to Minor Changes: In a fruit-box strategy puzzle, simply changing the word "lady" to "person" caused the model's accuracy to collapse, proving it relies on brittle linguistic associations.

Limitations and Future Work

The primary limitation is that the paper is a conceptual framework and a collection of examples rather than a massive, ready-to-use benchmark. The authors acknowledge that creating these "tough" schemas is difficult and time-consuming.

Future Work: The authors call for the community to build large-scale datasets based on these "extended" strategies. They conclude that solving these "Schemas from Hell" will require a fundamental shift in AI architecture—moving toward systems that can perform genuine, multi-step symbolic or epistemic reasoning.

Summary updated

Abstract

The Winograd Challenge has been advocated as a test of computer understanding with respect to commonsense reasoning. The challenge is based on Winograd Schemas: sentences that contain correferential ambiguities. Most Winograd Schemas are relatively easy for human subjects, and today the best computer systems for the Winograd Challenge can work close to human performance. In this paper, we examine the assumptions behind the Winograd Challenge, and investigate how far we can push the difficulty level of Winograd Schemas, proposing various strategies to build really challenging schemas.
BibTeX
@misc{cozman2020winograd,
  author = {Fábio Gagliardi Cozman and Hugo Munhoz},
  title = {{The Winograd Schemas from Hell}},
  year = {2020},
  doi = {10.5753/eniac.2020.12157},
}
Figure