The Referential Reader: A Recurrent Entity Network for Anaphora Resolution

Fei Liu, Luke Zettlemoyer, Jacob Eisenstein

Abstract

We present a new architecture for storing and accessing entity mentions during online text processing. While reading the text, entity references are identified, and may be stored by either updating or overwriting a cell in a fixed-length memory. The update operation implies coreference with the other mentions that are stored in the same cell; the overwrite operation causes these mentions to be forgotten. By encoding the memory operations as differentiable gates, it is possible to train the model end-to-end, using both a supervised anaphora resolution objective as well as a supplementary language modeling objective. Evaluation on a dataset of pronoun-name anaphora demonstrates strong performance with purely incremental text processing.

Introduction

Coreference resolution—the task of linking mentions of entities (like pronouns) to their antecedents—is a fundamental challenge in natural language processing. Most contemporary systems adopt a "mention-pair" approach, identifying all possible mentions in a document and classifying pairs. However, this method suffers from quadratic computational complexity and lacks cognitive plausibility, as humans resolve references incrementally while reading.

The "Referential Reader" (RefReader) proposes a novel, cognitively inspired architecture that performs anaphora resolution incrementally. By maintaining a fixed-size, differentiable memory of active entities, the model processes text from left to right, making coreference decisions on-the-fly.

Conceptual overview of the Referential Reader's memory operations during incremental reading.

Methodology

The RefReader architecture builds upon the Recurrent Entity Network framework, integrating a memory unit directly into a recurrent neural network.

The Recurrent Unit

The model utilizes a modified Gated Recurrent Unit (GRU) to maintain a hidden state $h_t$. Unlike standard GRUs, this unit integrates information from the memory unit $\mathcal{M}_t$ to inform its predictions, creating a feedback loop between linguistic context and entity state.

The Memory Unit

The memory consists of a fixed set of $N$ cells. Each cell stores a "key" representing the entity and a "value" representing its current state. For every token, the model employs differentiable gates to perform one of three operations:

  1. Update: The token refers to an existing entity in memory (coreference).
  2. Overwrite: The token introduces a new entity, replacing a less relevant one.
  3. Ignore: The token is not an entity mention.
The architectural flow of the Referential Reader, showing the interaction between input embeddings, the memory unit, and the hidden state.

Salience and Forgetting

To manage the fixed-size memory, the model tracks the "salience" of each cell. Salience increases when an entity is mentioned and decays over time. When a new entity must be stored in a full memory, a Gumbel-softmax distribution is used to select and "forget" the least salient cell, mimicking human-focus and memory decay.

Experimental Setup

The model was evaluated on the GAP (Gendered Ambiguous Pronouns) benchmark, which contains 8,908 ambiguous pronoun-name pairs extracted from Wikipedia. This dataset is specifically designed to measure gender bias in coreference systems. Additionally, a Wikipedia corpus of 3.8 million tokens was used for unsupervised language modeling pre-training, which helps the model learn entity relationships before fine-tuning on the coreference task.

Results and Discussion

The RefReader demonstrates that incremental processing is not only viable but highly effective.

  • Performance: The base RefReader achieved an F1 score of 72.1, surpassing strong heuristic baselines like Parallelism+URL (70.6).
  • BERT Integration: Incorporating BERT embeddings boosted the F1 score to 78.8, outperforming standard mention-pair models.
  • Efficiency: Remarkably, the model remains effective with as few as two memory cells, successfully "recycling" cells in longer documents.

A key insight is the synergy between language modeling and coreference resolution. Training the model to predict the next word provides a "warm start" for memory operations, as the model must track entities to maintain local coherence.

Conclusion

The Referential Reader offers a computationally efficient and cognitively plausible alternative to batch-processing coreference models. While future work is needed to scale the architecture to complex narratives with many active characters and to explore purely incremental transformer-based embeddings, the RefReader proves that a small, well-managed working memory is a powerful tool for natural language understanding.

Summary updated

Abstract

We present a new architecture for storing and accessing entity mentions during online text processing. While reading the text, entity references are identified, and may be stored by either updating or overwriting a cell in a fixed-length memory. The update operation implies coreference with the other mentions that are stored in the same cell; the overwrite operation causes these mentions to be forgotten. By encoding the memory operations as differentiable gates, it is possible to train the model end-to-end, using both a supervised anaphora resolution objective as well as a supplementary language modeling objective. Evaluation on a dataset of pronoun-name anaphora demonstrates strong performance with purely incremental text processing.
BibTeX
@inproceedings{liu2019referential,
  author = {Fei Liu and Luke Zettlemoyer and Jacob Eisenstein},
  title = {{The Referential Reader: A Recurrent Entity Network for Anaphora Resolution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=d2CVKJAO8VN},
}
Figure