Abstract: The Winograd Schema Challenge (WSC) dataset WSC273 and its inference counterpart WNLI are popular benchmarks for natural language understanding and commonsense reasoning. In this paper, we show that the performance of three language models on WSC273 strongly improves when fine-tuned on a similar pronoun disambiguation problem dataset (denoted WSCR). We additionally generate a large unsupervised WSC-like dataset. By fine-tuning the BERT language model both on the introduced and on the WSCR dataset, we achieve overall accuracies of 72.5% and 74.7% on WSC273 and WNLI, improving the previous state-of-the-art solutions by 8.8% and 9.6%, respectively. Furthermore, our fine-tuned models are also consistently more robust on the "complex" subsets of WSC273, introduced by Trichelair et al. (2018).
Introduction
The Winograd Schema Challenge (WSC) serves as a critical benchmark for evaluating whether AI systems possess true linguistic understanding or merely rely on statistical pattern matching. WSC tasks involve pairs of sentences differing by a single "special word" that alters the referent of a pronoun. For instance, in the sentence "The trophy didn’t fit into the suitcase because it was too [large/small]," the pronoun "it" refers to the trophy if the word is "large" and the suitcase if the word is "small."
Solving these schemas requires commonsense reasoning about physical properties and world knowledge. Historically, neural networks have struggled with WSC due to the small size of available datasets, which typically contain only a few hundred examples, leading to significant overfitting risks.
Methodology
To address the data scarcity inherent in the WSC, this research employs a multi-faceted fine-tuning strategy:
- Fine-tuning on WSCR: The authors utilize the Definite Pronoun Resolution (WSCR) dataset—a larger, structurally similar but less complex dataset—as a training foundation. While previous work suggested WSCR might not generalize to the harder WSC, this study demonstrates that when paired with the BERT architecture, it serves as a vital stepping stone.
- MaskedWiki Dataset: To further augment training data, the researchers developed a heuristic to automatically mine 2.4 million WSC-like examples from English Wikipedia. This unsupervised dataset, though noisy, provides the scale necessary for the model to learn structural nuances of pronoun referents.
- Margin-based Loss Function: Instead of standard language modeling objectives, a custom margin-based loss function was implemented. This function penalizes the model if the probability of the incorrect candidate is too close to the correct one, forcing a clearer distinction between choices.
Results
The proposed fine-tuning strategies yielded significant improvements across all evaluated benchmarks. As shown in the table below, the combination of BERT with the WSCR and MaskedWiki datasets (BERT_WIKI_WSCR) achieved a new state-of-the-art accuracy of 72.5% on WSC273 and 74.7% on WNLI.
Beyond overall accuracy, the models demonstrated improved robustness on complex subsets, including "switched" and "consistent" pairs. The "consistency" metric measures the model's ability to correctly resolve both versions of a Winograd pair. While the fine-tuned BERT models showed marked improvement (reaching 0.550 consistency), this remains a challenging area compared to human-level reasoning.
Discussion and Conclusion
The results indicate that generalization from simpler pronoun resolution tasks to complex commonsense reasoning is possible when using powerful pre-trained models like BERT. The sheer scale of the MaskedWiki dataset, despite its noise, helps the model internalize the structural nuances required for the task.
However, the "consistency gap" suggests that models may still rely on sophisticated pattern matching rather than the rock-solid logical reasoning humans employ. Future work will focus on refining synthetic data generation through better filtering to create "harder" training sets and further addressing the logical consistency of model predictions.
Summary updated