TL;DR
This paper introduces "xCoRe," a unified framework for coreference resolution that tackles the challenges of both very long documents and cross-document coreference. By framing these as a "cross-context" problem and proposing a novel three-step pipeline, xCoRe achieves state-of-the-art results on cross-document tasks and strong performance on long documents, offering a versatile solution for various coreference scenarios.
What Problem Does This Paper Address?
Coreference resolution (CR) is the task of identifying mentions in text that refer to the same real-world entity. While current CR systems excel at short to medium-sized texts, they struggle with two key challenges:
- Very Long Documents: Traditional neural models, often based on Transformers, face quadratic memory costs, making it difficult to process extremely long texts like books or lengthy articles. Existing solutions often involve splitting documents into smaller windows, which can lead to performance degradation.
- Cross-Document Coreference: Resolving coreference across multiple, distinct documents is even more challenging. Current state-of-the-art methods often require gold mentions (pre-identified entities) and struggle to scale effectively.
These two settings have typically been treated separately. This paper unifies them under the concept of "cross-context coreference" and proposes a single architecture, xCoRe, to handle all these scenarios efficiently.
Methods
xCoRe employs a novel three-step pipeline designed for "cross-context coreference":
- Within-Context Mention Extraction: This step identifies all potential mentions (spans of text referring to entities) within each individual text segment (context). It uses an end-to-end approach similar to Maverick, predicting mention starts and then their ends.
- Within-Context Mention Clustering: After extracting mentions, this stage groups them into local clusters within each context. This is based on established methods like LingMess, which use specialized scorers for different types of mention pairs (e.g., pronoun-pronoun, entity-pronoun).
- Cross-Context Cluster Merging: This is the core innovation. Instead of directly linking mentions across contexts, xCoRe merges the clusters identified in the previous step. It computes representations for each local cluster and then uses a linear classification layer to predict the probability of two clusters from different contexts referring to the same entity. This approach is designed to be invariant to the order of contexts, making it suitable for both sequential (long documents) and non-sequential (cross-document) settings.
The key novelty lies in this cluster-merging strategy, which avoids the quadratic complexity of directly comparing all mention pairs across contexts and allows for joint training on shared resources.
Datasets
The paper evaluates xCoRe on a variety of datasets covering different coreference settings:
- Cross-Document:
- ECB+: A well-established dataset of news articles organized into topics, used for evaluating entity coreference.
- SciCo: A dataset focused on scientific documents, linking mentions of scientific concepts.
- Long-Document:
- LitBank: A collection of 100 works of fiction, with an average document length of 2,000 tokens.
- Animal Farm: A manually annotated novel by George Orwell.
- BookCoref: A large-scale benchmark of automatically annotated books.
- Medium-Size:
- OntoNotes: A widely used, richly annotated corpus covering multiple genres.
- PreCo: A dataset of preschooler vocabulary for coreference resolution.
The paper does not introduce new datasets but leverages existing ones to demonstrate xCoRe's versatility.
Key Insights and Ideas
- Unifying Coreference Settings: Framing long-document and cross-document coreference as a single "cross-context" problem simplifies the architecture and enables joint training.
- Cluster Merging is Key: The paper argues that merging clusters rather than individual mentions across contexts is a more efficient and effective strategy, especially as the number of contexts increases.
- Leveraging Shared Data: The cross-context formulation allows xCoRe to benefit from training on combined long- and cross-document resources, increasing data availability and improving performance, particularly for the challenging cross-document task.
- End-to-End Solution: xCoRe provides a unified, end-to-end system that handles mention extraction, within-context clustering, and cross-context merging in a single pipeline, eliminating the need for separate modules or handcrafted features.
Key Results
- State-of-the-Art on Cross-Document: xCoRe achieves new state-of-the-art results on cross-document benchmarks like ECB+ and SciCo, significantly outperforming previous methods like PMCoref, especially when pre-trained on other datasets.
- Strong Long-Document Performance: xCoRe demonstrates robust performance on long-document benchmarks, surpassing several existing methods on Animal Farm and aligning closely with state-of-the-art on LitBank.
- Versatility: The model maintains top-tier results on traditional medium-size datasets like OntoNotes and PreCo, showcasing its generalization capabilities.
- Impact of Merging: The cluster merging strategy proves crucial, with performance gaps widening as the number of contexts increases, highlighting its effectiveness.
Limitations and Future Work
- English-Centric: The current experiments are limited to English entity coreference. The authors suggest that the model is language-agnostic and can be extended to event coreference with minor modifications, leaving this as future work.
- Hardware Constraints: Experiments were limited by available hardware (a single RTX 4090), impacting the maximum window size for long-document training and the benchmarking of very large autoregressive models.
- Future Work: The authors envision xCoRe benefiting downstream applications by providing a robust, end-to-end system for challenging coreference scenarios. Further exploration into other languages and event coreference is also suggested.
Summary updated