Abstract
Yuval Varkel, Amir Globerson. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Coreference resolution—the task of determining which mentions in a text refer to the same real-world entity—is a cornerstone of natural language understanding. However, it remains notoriously difficult to automate. Modern systems rely on "mention representations," vector embeddings that capture the essence of a span of text. While general-purpose embeddings like BERT or SpanBERT provide a strong foundation, their training objectives (such as predicting a missing word) are often "distant" from the specific logic required for coreference.
Furthermore, high-quality labeled data for coreference is expensive and requires expert annotators. This research explores a way to leverage vast amounts of unlabeled text to "warm up" a coreference model specifically for the task of identifying and clustering entities.
Methods: Targeted Self-Supervised Pre-training
The authors propose moving beyond general language model pre-training by introducing two tasks that directly mimic coreference challenges:
- Name Masking: This task exploits the heuristic that if a specific name (e.g., "Alice") appears multiple times in a document, those mentions are almost certainly coreferent. The authors identify these clusters in unlabeled text, mask some names with a
[MASK]token or a random word, and then train the model to cluster the masked mentions with the original, unmasked names based on context. - Pronoun Masking: Pronouns are the "glue" of coreference. The authors mask pronouns and train a small feed-forward network to predict the original pronoun (e.g., "he," "she," "it") using only the vector representation of that mention. This forces the representation to encode critical entity features like gender and number.
These tasks are integrated into a baseline model built on SpanBERT, which masks contiguous spans rather than individual words, making it a natural fit for representing entity mentions.
Experimental Setup
The model was pre-trained on a massive dump of English Wikipedia for 700,000 steps. Evaluation primarily focused on the GAP (Gendered Ambiguous Pronouns) dataset, a human-annotated corpus of 4,454 Wikipedia snippets designed to be gender-balanced and challenging for ambiguous pronoun resolution. Secondary evaluations were conducted on Ontonotes to test transferability to traditional coreference benchmarks.
Key Results
The proposed approach established a new state-of-the-art (SOTA) on the GAP dataset across two primary tasks.
Detected Mentions Task
In the Detected Mentions task, the model must identify mentions and resolve them. The pre-trained model achieved an 88.56 F1 score, a significant improvement over the SpanBERT Large baseline of 86.64.
Gold Two-Mention Task
When provided with "gold" (correct) mentions, the model achieved an F1 of 92.86, surpassing previous top-performing models such as Attree (2019) and Ionita et al. (2019).
Bias and Equity
By down-sampling masculine pronouns during pre-training, the authors maintained a healthy gender bias ratio ($F_1^F / F_1^M$) of 0.96 to 0.97, ensuring equitable performance across genders.
Discussion and Conclusion
The success of Name Masking demonstrates that the surrounding context of a mention often contains enough signal to identify an entity, even if the name is hidden. This research highlights the value of pre-training coreference-specific scoring functions rather than just the underlying transformer.
While results on the GAP dataset are impressive, gains on the larger Ontonotes dataset were more modest, suggesting that Wikipedia-style pre-training may need further tuning for diverse linguistic structures like news or conversation. Future work may involve developing more sophisticated self-supervised tasks involving entity aliases or cross-document coreference to further reduce reliance on human-labeled data.
Summary updated