Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference

Robert L. Logan Iv, Andrew McCallum, Sameer Singh, Daniel M. Bikel

Introduction

Cross-Document Entity Coreference (CDC) is the task of grouping mentions of entities across multiple documents into clusters representing unique real-world entities. Unlike Entity Linking (EL), which maps mentions to a static knowledge base like Wikipedia, CDC must discover entities dynamically. This is essential for processing real-world data streams—such as news feeds or research repositories—where new entities emerge constantly.

This paper introduces a new benchmark for Streaming CDC, focusing on systems that must process mentions incrementally with constant time and memory constraints. The authors demonstrate that by combining traditional feature-based encoders with modern neural models and employing intelligent memory management, systems can achieve high accuracy while storing only a small fraction of observed mentions.

Overview of the streaming CDC process where mentions arrive sequentially and are clustered into entities.

Methodology

The proposed streaming CDC pipeline consists of two primary stages: mention encoding and incremental clustering.

Mention Encoding

Each mention and its surrounding context must be transformed into a vector representation. The authors compared three distinct approaches:

  • Feature-based: Traditional methods using character skip-bigrams and tf-idf to capture surface-level text similarity.
  • Neural: Modern BERT-based encoders and specialized entity linkers like RELIC and BLINK that capture deep contextual meaning.
  • Hybrid: A combination of feature-based representations (for the entity name) and neural representations (for the context).
Example of a mention $m_1$ with its surrounding context used for encoding.

Incremental Clustering

As mentions arrive, they must be assigned to existing clusters or used to form new ones. The study evaluates:

  • Greedy Nearest Neighbor (GNN): A fast, simple approach that clusters a mention with its most similar neighbor if a similarity threshold is met.
  • GRINCH: A sophisticated hierarchical clustering algorithm capable of "rearranging" its tree structure to fix earlier mistakes. The authors introduced a Bounded GRINCH variant that prunes the tree to maintain a fixed memory limit.

Memory Management and Scalability

A core challenge in streaming is the linear growth of mention history. To maintain constant memory, the authors implemented policies like Window, Cache, and Diversity-Cache. The Diversity-Cache policy, which retains a diverse set of mentions rather than just the most recent ones, allowed models to maintain near-peak performance while caching only 10% of total mentions.

To quantify this, the authors introduced the Maximum Active Entities (MAE) metric, representing the minimum number of mentions a system must store to achieve perfect clustering for a specific dataset.

Experimental Results and Insights

The benchmark was tested across three diverse datasets: AIDA CoNLL-YAGO (News), MedMentions (Biomedical), and Zeshel (Zero-Shot).

Key Findings

  • Hybrid Models Excel: Surprisingly, pure neural models often struggled with surface-form matching. Hybrid encoders achieved the best results (e.g., 91.1 CoNLL F1 on AIDA) by leveraging both name-based features and neural context.
  • The Power of Pruning: Increasing memory has diminishing returns. Systems restricted to a memory footprint proportional to the number of unique entities performed within 2% of unbounded systems.
  • Domain Sensitivity: The choice of clustering algorithm is domain-dependent. GNN worked best for News, while the complex hierarchical structure of GRINCH performed better on technical biomedical data.

Conclusion and Future Work

This research establishes a rigorous framework for evaluating streaming CDC systems. The results highlight that high-performance coreference is possible in a streaming setting through hybrid encoding and smart memory pruning. Future directions include exploring end-to-end systems that do not rely on pre-defined "gold" mention boundaries and developing better ways to integrate surface features directly into neural architectures.

Summary updated

BibTeX
@inproceedings{iv2021benchmarking,
  author = {Robert L. Logan IV and Andrew McCallum and Sameer Singh and Daniel M. Bikel},
  title = {{Benchmarking Scalable Methods for Streaming Cross Document Entity Coreference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.364},
}
Figure