WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

Vid Kocijan, Oana-Maria Camburu, Ana-Maria Cretu, Yordan Yordanov, Phil Blunsom, Thomas Lukasiewicz

Abstract

Pronoun resolution is a major area of natural language understanding. However, large-scale training sets are still scarce, since manually labelling data is costly. In this work, we introduce WikiCREM (Wikipedia CoREferences Masked) a large-scale, yet accurate dataset of pronoun disambiguation instances. We use a language-model-based approach for pronoun resolution in combination with our WikiCREM dataset. We compare a series of models on a collection of diverse and challenging coreference resolution problems, where we match or outperform previous state-of-the-art approaches on 6 out of 7 datasets, such as GAP, DPR, WNLI, PDP, WinoBias, and WinoGender. We release our model to be used off-the-shelf for solving pronoun disambiguation.

Introduction

Coreference resolution, particularly pronoun resolution, involves identifying which entity a pronoun refers to in a given context. While seemingly intuitive for humans, it remains a significant challenge for Natural Language Processing (NLP) systems. "Hard" pronoun resolution cases—those requiring world knowledge or deep contextual understanding rather than simple syntactic cues—are especially difficult. For instance, in the sentence "The trophy didn't fit into the brown suitcase because it was too large," determining whether "it" refers to the trophy or the suitcase requires semantic reasoning.

Modern neural models like BERT require vast amounts of data to master these nuances. However, high-quality, manually labeled datasets for coreference resolution are expensive to produce and typically small. This data scarcity creates a bottleneck for training robust pronoun resolvers.

The WikiCREM Methodology

To overcome the data bottleneck, we developed WikiCREM (Wikipedia CoREferences Masked), a massive unsupervised dataset comprising 2.4 million examples. The generation process leverages English Wikipedia through a three-step automated pipeline:

  1. Entity Identification: Passages are identified where a personal name appears at least twice.
  2. Masking and Distraction: A subsequent occurrence of the name is masked. To ensure the task is challenging, at least one other distinct personal name must appear earlier in the text to serve as a "distractor."
  3. Resolution Task: The model is tasked with predicting which of the preceding names belongs in the [MASK] slot.

By training the model to distinguish between names in context, we effectively teach it the semantic associations required for pronoun resolution. This "name-masking" serves as a high-fidelity proxy for pronoun disambiguation. Furthermore, WikiCREM examples are significantly more natural than previous automated approaches; 63% of WikiCREM instances sound natural when the name is replaced by a pronoun, compared to just 7% in earlier datasets like MaskedWiki.

Experimental Setup

We utilized BERT-large as our base model. The training objective combined standard Negative Log-Likelihood (NLL) with a Max-Margin term. This combined loss encourages the model to not only identify the correct entity but to do so with high confidence relative to distractors.

Models were evaluated across seven major benchmarks:

  • GAP: Gender-balanced Wikipedia snippets.
  • DPR: Definite Pronoun Resolution.
  • WSC / WNLI / PDP: Logic-heavy Winograd Schema Challenge variants.
  • WinoBias / WinoGender: Diagnostics for gender bias.

Results and Discussion

The inclusion of WikiCREM data led to substantial performance gains, with our models achieving state-of-the-art (SOTA) results on 6 of the 7 evaluated datasets.

Performance of BERT models fine-tuned on WikiCREM across various coreference resolution benchmarks.

As shown in the table above, BERT_WikiCREM variants consistently outperform standard BERT baselines and previous SOTA methods. Notable improvements include a 5.9% gain on GAP, an 8.4% boost on DPR, and a significant 12.7% jump on PDP.

Beyond raw accuracy, WikiCREM also aids in bias mitigation. Despite the inherent male skew in Wikipedia names, models trained on WikiCREM showed reduced bias on gender diagnostic datasets like WinoBias. This suggests that the large-scale semantic signal provided by the dataset helps the model learn more robust, less stereotypical associations.

Conclusion

WikiCREM demonstrates that large-scale unsupervised data can effectively bridge the gap where human-labeled data is scarce. By using name-masking as a proxy for pronoun resolution, we can generate millions of high-quality training examples that significantly improve the performance of transformer-based models. Future work may involve expanding this technique beyond personal names to include common nouns, potentially creating an even more comprehensive corpus for general coreference resolution.

Summary updated

Abstract

Pronoun resolution is a major area of natural language understanding. However, large-scale training sets are still scarce, since manually labelling data is costly. In this work, we introduce WikiCREM (Wikipedia CoREferences Masked) a large-scale, yet accurate dataset of pronoun disambiguation instances. We use a language-model-based approach for pronoun resolution in combination with our WikiCREM dataset. We compare a series of models on a collection of diverse and challenging coreference resolution problems, where we match or outperform previous state-of-the-art approaches on 6 out of 7 datasets, such as GAP, DPR, WNLI, PDP, WinoBias, and WinoGender. We release our model to be used off-the-shelf for solving pronoun disambiguation.
BibTeX
@inproceedings{kocijan2019wikicrem,
  author = {Vid Kocijan and Oana-Maria Camburu and Ana-Maria Cretu and Yordan Yordanov and Phil Blunsom and Thomas Lukasiewicz},
  title = {{WikiCREM: A Large Unsupervised Corpus for Coreference Resolution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=vaxExh8fl-F},
}

Loading related content...

Figure