The Referential Reader: A Recurrent Entity Network for Anaphora Resolution

Fei Liu, Luke Zettlemoyer, Jacob Eisenstein

Abstract

We present a new architecture for storing and accessing entity mentions during online text processing. While reading the text, entity references are identified, and may be stored by either updating or overwriting a cell in a fixed-length memory. The update operation implies coreference with the other mentions that are stored in the same cell; the overwrite operation causes these mentions to be forgotten. By encoding the memory operations as differentiable gates, it is possible to train the model end-to-end, using both a supervised anaphora resolution objective as well as a supplementary language modeling objective. Evaluation on a dataset of pronoun-name anaphora demonstrates strong performance with purely incremental text processing.

Introduction

Traditional coreference resolution systems typically employ a "mention-pair" approach, evaluating every possible pair of nouns and pronouns within a document. While accurate, this method is computationally expensive—scaling quadratically with the number of mentions—and cognitively implausible. Humans do not wait until the end of a text to resolve references; we process language incrementally, linking pronouns to entities "online" as we read.

The Referential Reader addresses this challenge by introducing an incremental model that tracks entities in a fixed-size working memory. It decides in real-time whether to link a new word to a known entity, overwrite an old memory with a new entity, or disregard the token if it does not refer to a tracked entity.

Methods

The architecture of the Referential Reader is designed to maintain a persistent but finite representation of entities encountered during text processing. It consists of two primary components: a Recurrent Unit and a Memory Unit.

Architecture Overview

The Recurrent Unit is a modified Gated Recurrent Unit (GRU) that maintains a hidden state $h_t$. This unit integrates information from the current input $x_t$, a "pre-recurrent" state $\tilde{h}_t$, and the current state of the entity memory $\mathcal{M}_t$. This hierarchical structure allows the model to maintain local context while simultaneously accessing long-term entity information.

Architecture of the Referential Reader, showing the flow from input embeddings to the memory unit and hidden state.

The Memory Unit and Operations

The Memory Unit consists of $N$ cells, each tracking a specific entity. Each cell contains a key (representing the entity's identity), a value (its current state), and a salience score (representing its importance or recency). As the model reads each token, it uses differentiable gates to perform three possible actions:

  • Update: If a token refers to an existing entity, the corresponding memory cell is updated, and its salience score increases.
  • Overwrite: When a new entity is introduced, the model uses a Gumbel-softmax distribution to select the least salient memory cell to overwrite.
  • Disregard: If the token is not an entity mention, the memory remains largely unchanged, though salience scores naturally decay over time.
Illustration of memory operations (update and overwrite) during incremental text processing for the sentence "Ismael told Captain Ahab he saw Moby-Dick".

Semi-Supervised Training

A key feature of the model is its ability to be trained in a semi-supervised manner. While it can be trained on standard coreference data (like the GAP benchmark), it can also be pre-trained on massive unlabeled corpora using a language modeling objective (predicting the next word). This allows the model to learn latent entity structures and behaviors from context even without explicit coreference labels.

Results and Discussion

The model was evaluated on the GAP (Gendered Ambiguous Pronouns) benchmark, which consists of thousands of pronoun-name pairs from Wikipedia designed to be gender-balanced and challenging.

Key Performance Metrics

  • State-of-the-Art Performance: The Referential Reader achieved an F1 score of 72.1 on the GAP test set, outperforming strong heuristic baselines. When integrated with BERT embeddings, performance reached 78.8 F1.
  • Low Gender Bias: The model maintained a gender bias ratio of 0.98 (where 1.0 is perfect parity), demonstrating that the architecture does not rely on gendered stereotypes to resolve references.
  • Salience as a Shield: Analysis shows that salience effectively protects important entities. As an entity is mentioned more frequently, its salience increases, making the model less likely to overwrite that cell with new, potentially irrelevant information.

Conclusion

The Referential Reader demonstrates that incremental, memory-based entity tracking is a viable and efficient alternative to traditional mention-pair models. By processing text left-to-right with a fixed memory size, it avoids quadratic complexity and more closely mimics human linguistic processing. Future work will explore scaling the model to longer, more complex documents and incorporating additional morphosyntactic constraints to further refine the gating mechanisms.

Summary updated

Abstract

We present a new architecture for storing and accessing entity mentions during online text processing. While reading the text, entity references are identified, and may be stored by either updating or overwriting a cell in a fixed-length memory. The update operation implies coreference with the other mentions that are stored in the same cell; the overwrite operation causes these mentions to be forgotten. By encoding the memory operations as differentiable gates, it is possible to train the model end-to-end, using both a supervised anaphora resolution objective as well as a supplementary language modeling objective. Evaluation on a dataset of pronoun-name anaphora demonstrates strong performance with purely incremental text processing.
BibTeX
@inproceedings{liu2019referential,
  author = {Fei Liu and Luke Zettlemoyer and Jacob Eisenstein},
  title = {{The Referential Reader: A Recurrent Entity Network for Anaphora Resolution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=d2CVKJAO8VN},
}
Figure