Incremental Neural Coreference Resolution in Constant Memory

Patrick Xia, João Sedoc, Benjamin Van DurmeEMNLP 2020 · OpenReview

Abstract

Patrick Xia, João Sedoc, Benjamin Van Durme. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

Coreference resolution—the task of determining which words and phrases (mentions) in a text refer to the same real-world entity—is a fundamental challenge in NLP. While modern deep learning models have become incredibly accurate, they face a significant "memory problem": most require processing the entire document at once, leading to memory usage that explodes as documents get longer. This makes processing long-form text like novels or legal documents computationally prohibitive.

In their EMNLP 2020 paper, Xia et al. present an incremental model that achieves state-of-the-art performance while using constant memory, regardless of document length.

Methodology

The researchers propose an Incremental Neural Coreference system that transforms the task into an incremental clustering algorithm.

Incremental Clustering and Entity Representations

Instead of comparing spans to every other span in an $O(n^2)$ fashion, the model compares spans to entities. As the model reads a document segment by segment, it identifies potential mentions. For each mention, it decides whether to link it to an existing entity "cluster" or to initialize a new one.

Crucially, the model maintains a single vector representation for each entity. When a new mention is linked, the entity's vector is updated using a learned weighted average. This allows the entity's "meaning" to evolve as the narrative progresses without needing to store every individual mention embedding.

Constant Memory via Eviction

To keep memory usage from growing, the model employs an "eviction policy." It tracks the salience of entities based on frequency and recency. When memory capacity is reached, less salient entities (such as singletons mentioned once long ago) are moved out of active GPU memory. This approach reduces memory complexity from $O(n)$ or $O(n^2)$ to $O(1)$ relative to document length.

Experimental Results

The model was evaluated on the OntoNotes 5.0 benchmark, the standard for coreference resolution.

Performance Comparison

On the OntoNotes 5.0 test set, the model achieved an Avg. F1 of 79.4, compared to the baseline's 79.6. This remarkably small trade-off demonstrates that exhaustive pairwise comparison is not strictly necessary for high performance.

Comparison of performance metrics on the OntoNotes 5.0 test set.

Memory Efficiency

The primary advantage of the proposed model is its memory efficiency. While baseline models (like JS-L) see memory usage grow linearly or quadratically—often exceeding 11GB for long documents—the incremental model stays flat at approximately 2.0 GB of GPU memory, regardless of document length.

Peak memory usage as a function of document length. The proposed model (Ours) maintains constant memory usage compared to growing baselines.

Discussion

The authors found that evicting "singleton" clusters after a certain distance had almost zero impact on accuracy but significantly stabilized memory usage. This suggests that long-range coreference is often about maintaining a consistent entity state rather than exhaustive comparison.

However, limitations remain. The current model does not use speaker metadata, leading to a performance dip in conversational data. Future work could involve a "learned" eviction policy where the model dynamically determines which entities are no longer relevant to the narrative.

Conclusion

By mimicking human-like incremental processing and limited working memory, Xia et al. (2020) provide a scalable solution for coreference resolution in long documents. Their approach achieves competitive accuracy while maintaining a constant, low memory footprint, paving the way for more efficient NLP on large-scale texts.

Summary updated

BibTeX
@inproceedings{xia2020incremental,
  author = {Patrick Xia and João Sedoc and Benjamin Van Durme},
  title = {{Incremental Neural Coreference Resolution in Constant Memory}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=7YbP2kMQXWJ},
}
Figure