Abstract
We present a new architecture for storing and accessing entity mentions during online text processing. While reading the text, entity references are identified, and may be stored by either updating or overwriting a cell in a fixed-length memory. The update operation implies coreference with the other mentions that are stored in the same cell; the overwrite operation causes these mentions to be forgotten. By encoding the memory operations as differentiable gates, it is possible to train the model end-to-end, using both a supervised anaphora resolution objective as well as a supplementary language modeling objective. Evaluation on a dataset of pronoun-name anaphora demonstrates strong performance with purely incremental text processing.
Introduction
Coreference resolution—the task of linking mentions of entities (like pronouns) to their antecedents—is a fundamental challenge in natural language processing. Most contemporary systems adopt a "mention-pair" approach, identifying all possible mentions in a document and classifying pairs. However, this method suffers from quadratic computational complexity and lacks cognitive plausibility, as humans resolve references incrementally while reading.
The "Referential Reader" (RefReader) proposes a novel, cognitively inspired architecture that performs anaphora resolution incrementally. By maintaining a fixed-size, differentiable memory of active entities, the model processes text from left to right, making coreference decisions on-the-fly.
Methodology
The RefReader architecture builds upon the Recurrent Entity Network framework, integrating a memory unit directly into a recurrent neural network.
The Recurrent Unit
The model utilizes a modified Gated Recurrent Unit (GRU) to maintain a hidden state $h_t$. Unlike standard GRUs, this unit integrates information from the memory unit $\mathcal{M}_t$ to inform its predictions, creating a feedback loop between linguistic context and entity state.
The Memory Unit
The memory consists of a fixed set of $N$ cells. Each cell stores a "key" representing the entity and a "value" representing its current state. For every token, the model employs differentiable gates to perform one of three operations:
- Update: The token refers to an existing entity in memory (coreference).
- Overwrite: The token introduces a new entity, replacing a less relevant one.
- Ignore: The token is not an entity mention.
Salience and Forgetting
To manage the fixed-size memory, the model tracks the "salience" of each cell. Salience increases when an entity is mentioned and decays over time. When a new entity must be stored in a full memory, a Gumbel-softmax distribution is used to select and "forget" the least salient cell, mimicking human-focus and memory decay.
Experimental Setup
The model was evaluated on the GAP (Gendered Ambiguous Pronouns) benchmark, which contains 8,908 ambiguous pronoun-name pairs extracted from Wikipedia. This dataset is specifically designed to measure gender bias in coreference systems. Additionally, a Wikipedia corpus of 3.8 million tokens was used for unsupervised language modeling pre-training, which helps the model learn entity relationships before fine-tuning on the coreference task.
Results and Discussion
The RefReader demonstrates that incremental processing is not only viable but highly effective.
- Performance: The base RefReader achieved an F1 score of 72.1, surpassing strong heuristic baselines like Parallelism+URL (70.6).
- BERT Integration: Incorporating BERT embeddings boosted the F1 score to 78.8, outperforming standard mention-pair models.
- Efficiency: Remarkably, the model remains effective with as few as two memory cells, successfully "recycling" cells in longer documents.
A key insight is the synergy between language modeling and coreference resolution. Training the model to predict the next word provides a "warm start" for memory operations, as the model must track entities to maintain local coherence.
Conclusion
The Referential Reader offers a computationally efficient and cognitively plausible alternative to batch-processing coreference models. While future work is needed to scale the architecture to complex narratives with many active characters and to explore purely incremental transformer-based embeddings, the RefReader proves that a small, well-managed working memory is a powerful tool for natural language understanding.
Summary updated