Coreference Resolution with Entity Equalization

Ben Kantor, Amir GlobersonACL 2019 · OpenReview

Abstract

This paper introduces "Entity Equalization," a method that represents every mention in a cluster as the sum of all its members, allowing the model to leverage global entity-level information. By combining this differentiable clustering approach with the first-ever application of BERT to coreference resolution, the authors achieved a new state-of-the-art result on the CoNLL-2012 benchmark, improving the average F1 score by 3.6%.

Introduction

Coreference resolution is the task of identifying all expressions (mentions) in a text that refer to the same real-world entity—for example, linking "John," "he," and "the baker" within a single document.

The primary challenge is that information about an entity is often fragmented. A single pronoun like "he" carries very little semantic information on its own. To accurately decide if "he" refers to "John," a model needs to look at the "global" properties of the entire entity cluster. While previous models used pairwise scoring or "antecedent averaging," they often missed the full context of the entity cluster. The authors sought a way to create global representations that are updated as the model learns, within an end-to-end trainable system.

Methodology

The authors build upon the end-to-end architecture of Lee et al. (2018) but introduce two major innovations:

Entity Equalization (EE)

Instead of just looking at previous mentions (antecedents), the EE mechanism posits that every mention in a cluster should be represented by the sum of all mentions in that cluster. Because the actual clustering process isn't naturally differentiable, they use a differentiable soft-clustering relaxation. This allows the model to calculate the probability of a mention belonging to a specific entity and use those probabilities to "equalize" the representations across the cluster.

The soft-clustering matrix illustrates how mentions (rows) are assigned to entity clusters (columns) to enable equalization.

Convolutional BERT

This paper marks the first use of BERT for coreference. Since BERT has a fixed sequence length (512 tokens) and coreference documents are often much longer, the authors applied BERT in a "convolutional" manner. They used a sliding window around each token to generate embeddings, then took a weighted average of the last four BERT layers to create the final token representation.

Results

The model was evaluated on the English portion of the OntoNotes 5.0 dataset (CoNLL-2012 Shared Task). The authors compared their model against the previous state-of-the-art (Lee et al., 2018).

Detailed performance metrics across MUC, B³, and CEAFφ₄ show the consistent gains provided by BERT and Entity Equalization.

The jump from 73.0% to 76.61% represents a 3.6% absolute improvement, a substantial margin for this task. While BERT contributed significantly to the performance gain, the Entity Equalization (EE) provided a consistent boost across all evaluation metrics, particularly in recall and F1 scores.

Key Insights and Discussion

  • Summation vs. Averaging: The authors argue that summing all mentions in a cluster provides a more robust "global" signature of the entity than simple averaging.
  • Differentiable Clusters: By treating the entity-to-mention mapping as a soft probability distribution, the entire global reasoning process becomes part of the neural network's gradient flow.
  • BERT's Superiority: Replacing ELMo embeddings with BERT provided the largest single performance jump. Even with a relatively small sliding window, BERT’s contextual representations proved much more powerful for resolving identity.
  • Refinement: The "Equalization" step acts as a refinement process. The model first makes an initial guess at the clusters, uses those clusters to update the mention features, and then makes a more informed final prediction.

Limitations and Future Work

Using BERT in a sliding-window fashion is computationally intensive, requiring the authors to limit window sizes due to memory constraints. Additionally, while EE provides global info within a document, it does not yet account for cross-document coreference. Future work could incorporate more structured entity attributes (like gender or semantic roles) or model the relationships between different entities.

Summary updated

BibTeX
@inproceedings{kantor2019coreference,
  author = {Ben Kantor and Amir Globerson},
  title = {{Coreference Resolution with Entity Equalization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=0_L0-r8h0w0},
}
Figure