WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

Vid Kocijan, Oana-Maria Camburu, Ana-Maria Cretu, Yordan Yordanov, Phil Blunsom, Thomas Lukasiewicz

Abstract

Vid Kocijan, Oana-Maria Camburu, Ana-Maria Cretu, Yordan Yordanov, Phil Blunsom, Thomas Lukasiewicz. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Pronoun resolution—the task of determining which entity a pronoun like "he," "she," or "it" refers to—is a cornerstone of natural language understanding. While humans perform this task effortlessly, it remains a significant challenge for NLP models, particularly when the resolution cannot be deduced from simple syntax.

The primary bottleneck in this field has been data scarcity. Modern neural models like BERT are data-hungry, yet manually labeling "hard" coreference examples is expensive and slow. This paper introduces WikiCREM, a massive unsupervised dataset of 2.4 million examples designed to train large-scale models without the need for human annotators.

Methodology: The WikiCREM Approach

The authors propose a clever, unsupervised "masking" trick to generate data from English Wikipedia. The process follows these steps:

  1. Identify Passages: Find text where a personal name appears at least twice.
  2. Masking: Mask a non-first occurrence of that name.
  3. Distractors: Require at least one other distinct personal name to appear before the masked name to ensure the task is non-trivial.
  4. The Task: The model must predict which of the two names belongs in the [MASK] slot.

This approach effectively mimics the structure of pronoun resolution by using names as proxies. The authors fine-tuned BERT-Large using a specialized loss function that combines standard negative log-likelihood with a max-margin loss, forcing the model to prefer the correct name over distractors by a specific confidence margin.

Evaluation and Key Results

WikiCREM was evaluated against seven major coreference benchmarks. The results demonstrate that fine-tuning on this synthetic but high-quality data allows BERT to match or outperform previous state-of-the-art (SOTA) results on 6 out of 7 datasets.

Performance of WikiCREM-enhanced BERT on GAP, DPR, WSC, and WNLI benchmarks.

As shown in the table above, the WikiCREM-enhanced model achieved a 78.0% F1 score on the GAP dataset, a significant jump over the previous 72.1% SOTA. It also matched the SOTA on WNLI (74.7%) and showed substantial improvements on the DPR dataset (84.8%).

Bias Mitigation and Robustness

A critical contribution of WikiCREM is its impact on model fairness. Because the dataset is large and derived from diverse Wikipedia entries, it helps regularize the model against gender biases often found in smaller, human-curated datasets.

Performance on WinoBias (WB), Winogender, and PDP datasets, highlighting bias reduction and accuracy.

On diagnostic datasets like Winogender and WinoBias (WB), models trained with WikiCREM showed both higher overall accuracy and a marked reduction in performance gaps between genders. Furthermore, the model reached a new SOTA of 86.7% on the PDP dataset. Manual annotation confirmed that 63% of WikiCREM examples are "natural," meaning they sound like real-world sentences if the name is replaced by a pronoun—a massive improvement over the 7% naturalness of previous unsupervised methods.

Conclusion

WikiCREM demonstrates that large-scale, unsupervised data generation can effectively solve the data scarcity problem in coreference resolution. By leveraging the inherent structure of Wikipedia, the authors created a high-quality corpus that pushes the state-of-the-art while simultaneously addressing critical issues like gender bias in NLP models.

Summary updated

BibTeX
@inproceedings{kocijan2019wikicrem,
  author = {Vid Kocijan and Oana-Maria Camburu and Ana-Maria Cretu and Yordan Yordanov and Phil Blunsom and Thomas Lukasiewicz},
  title = {{WikiCREM: A Large Unsupervised Corpus for Coreference Resolution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=vaxExh8fl-F},
}
Figure