A Surprisingly Robust Trick for the Winograd Schema Challenge

Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, Thomas Lukasiewicz

Abstract

The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. In this paper, we show that the performance of three language models on WSC273 consistently and robustly improves when fine-tuned on a similar pronoun disambiguation problem dataset (denoted WSCR). We additionally generate a large unsupervised WSC-like dataset. By fine-tuning the BERT language model both on the introduced and on the WSCR dataset, we achieve overall accuracies of 72.5% and 74.7% on WSC273 and WNLI, improving the previous state-of-the-art solutions by 8.8% and 9.6%, respectively. Furthermore, our fine-tuned models are also consistently more accurate on the “complex” subsets of WSC273, introduced by Trichelair et al. (2018).

Introduction

The Winograd Schema Challenge (WSC) is a classic test of an AI's ability to use commonsense reasoning. It consists of sentence pairs that differ by only one "special word," which changes the referent of a pronoun. For example:

  • The trophy didn’t fit into the suitcase because it was too [large/small].
  • If the word is "large," it refers to the trophy. If the word is "small," it refers to the suitcase.

To solve this, a model cannot rely solely on grammar; it requires an understanding of physical properties and world knowledge. Historically, the primary obstacle for deep learning models has been data scarcity. The standard WSC evaluation set (WSC273) is small, making it difficult to train large neural networks without overfitting. This research demonstrates that fine-tuning on related datasets, previously thought to be ineffective, actually yields significant gains when applied to modern transformer-based models like BERT.

Methodology

The researchers employed a two-step fine-tuning approach using BERT-large, focusing on three key components:

  1. Masked Token Prediction: WSC is framed as a "fill-in-the-blank" problem rather than simple classification. The pronoun is masked, and the model calculates the probability of each candidate noun (e.g., "trophy" vs. "suitcase") filling that slot.
  2. Margin-Based Loss: An objective function is used that penalizes the model if the log-probability of the correct candidate does not exceed the incorrect one by a specific margin ($\beta$).
  3. Synthetic Data Generation (MaskedWiki): To address data scarcity, a massive dataset of 2.4 million examples was automatically generated from English Wikipedia. By identifying sentences with two occurrences of the same noun and masking the second, the authors created a large-scale, self-supervised proxy for WSC tasks.

Datasets

The study utilizes several key benchmarks and training sets:

  • WSC273: The gold-standard evaluation set for the Winograd Schema Challenge.
  • WNLI (Winograd NLI): An entailment-based version of WSC included in the GLUE benchmark.
  • WSCR (Winograd Schema Challenge Relaxed): A larger dataset (approx. 1,800 examples) containing simpler pronoun resolution problems.
  • MaskedWiki: The newly introduced 2.4M example dataset generated from Wikipedia for large-scale pre-training.

Results

The proposed model, BERT_WIKI_WSCR, established new state-of-the-art records. It achieved a 72.5% accuracy on WSC273 (an 8.8% improvement) and 74.7% on WNLI (a 9.6% improvement). Notably, this is the first model to significantly surpass the 65.1% "majority class" baseline on WNLI.

Table 1: Performance comparison of various models on WSC273, its subsets, and WNLI.

A key finding was the "robustness" of the fine-tuning. Contrary to earlier assumptions, fine-tuning on the "easier" WSCR dataset significantly helped BERT generalize to the harder WSC273. Furthermore, the synergy between the scale of MaskedWiki and the quality of WSCR was crucial; MaskedWiki "warmed up" the model to the task structure, while WSCR provided the necessary reasoning nuance.

Discussion and Conclusion

While the results represent a significant leap forward, challenges remain. The MaskedWiki dataset contains inherent noise, with approximately 8.5% of samples being unsolvable even for humans. Additionally, the 72.5% accuracy on WSC273 still trails human performance, which is near 100%. Future work may focus on better data-filtering techniques for synthetic data and applying this "masked-replacement" fine-tuning strategy to other data-scarce NLP tasks. This research highlights that with the right fine-tuning strategy, large language models can capture the underlying logic required for complex commonsense reasoning.

Summary updated

Abstract

The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. In this paper, we show that the performance of three language models on WSC273 consistently and robustly improves when fine-tuned on a similar pronoun disambiguation problem dataset (denoted WSCR). We additionally generate a large unsupervised WSC-like dataset. By fine-tuning the BERT language model both on the introduced and on the WSCR dataset, we achieve overall accuracies of 72.5% and 74.7% on WSC273 and WNLI, improving the previous state-of-the-art solutions by 8.8% and 9.6%, respectively. Furthermore, our fine-tuned models are also consistently more accurate on the “complex” subsets of WSC273, introduced by Trichelair et al. (2018).
BibTeX
@inproceedings{kocijan2019surprisingly,
  author = {Vid Kocijan and Ana-Maria Cretu and Oana-Maria Camburu and Yordan Yordanov and Thomas Lukasiewicz},
  title = {{A Surprisingly Robust Trick for the Winograd Schema Challenge}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=Qlf6y9GYvU},
}
Figure