Error-Driven Analysis of Challenges in Coreference Resolution

Jonathan K. Kummerfeld, Dan KleinEMNLP 2013 · OpenReview

Abstract

Coreference resolution metrics quantify errors but do not analyze them. Here, we consider an automated method of categorizing errors in the output of a coreference system into intuitive underlying error types. Using this tool, we first compare the error distributions across a large set of systems, then analyze common errors across the top ten systems, empirically characterizing the major unsolved challenges of the coreference resolution task.

Introduction

Coreference resolution—the task of determining which mentions in a text (e.g., "President Clinton," "he," and "Mr. Clinton") refer to the same real-world entity—is a cornerstone of natural language processing. Traditionally, researchers have relied on aggregate metrics like MUC, $B^3$, and CEAF to evaluate performance. However, these metrics provide a single scalar value that indicates the magnitude of failure without diagnosing the nature of the errors.

Is a system struggling with pronouns? Is it accidentally merging distinct entities? Or is it merely misidentifying phrase boundaries (e.g., "Gorbachev" vs. "Soviet leader Gorbachev")? Without a granular breakdown of error types, it is difficult to identify specific model weaknesses. This paper introduces an automated error-analysis tool to address this diagnostic gap.

Methodology

The proposed tool operates solely on the final output of any coreference system, requiring no internal logs or manual labor. It utilizes a transformation process that converts system-predicted clusters into "gold" (correct) clusters through a sequence of five algorithmic operations:

  1. Alter Span: Adjusts mention boundaries to match gold spans.
  2. Split: Separates incorrectly merged entities.
  3. Remove: Deletes spurious mentions.
  4. Introduce: Adds missing mentions.
  5. Merge: Combines entities that were incorrectly kept separate.
The five-step transformation process used to map system outputs to gold entities.

These operations are then mapped to seven intuitive error categories: Span Error, Missing Entity, Extra Entity, Missing Mention, Extra Mention, Divided Entity, and Conflated Entities. This mapping allows for a qualitative understanding of system failures.

Experimental Setup

The study primarily utilizes the CoNLL 2011 shared task data, based on the OntoNotes corpus, which includes news, magazine articles, and broadcast conversations. The authors evaluated outputs from all CoNLL 2011 participants and several state-of-the-art systems, including BART, UIUC, and Stanford’s coreference system.

Analysis and Insights

The analysis revealed several key characteristics of current coreference systems:

  • Metric Leaders vs. Error Leaders: Top-performing systems on aggregate metrics are not necessarily superior in every error category. A system with a high overall F-score might still struggle significantly with specific linguistic phenomena, such as pronoun resolution, compared to lower-ranked systems.
  • Superficial Span Errors: Approximately 27% of span errors do not correspond to valid nodes in a parse tree. These often result from minor disagreements in annotation (e.g., including a possessive 's') rather than deep linguistic failures.
  • The "Small Entity" Penalty: Missing entities containing only a few mentions cause the most significant drop in recall. Systems frequently overlook these small entities entirely, leading to a substantial hit to standard metrics.
  • Pronoun Conflation and Division: Systems often fail to link pronouns to their correct antecedents. This leads to "Divided Entities," where a single person is split into two entities (e.g., the name in one cluster and the pronoun in another).
Examples of operations and their corresponding error types in coreference resolution.

Results

Quantifying the impact of fixing each error type showed that Divided Entities are the most frequent error across all systems. However, fixing Missing Entities provides the most substantial boost to F-scores (e.g., a ~10 point jump in MUC F1 for some systems), identifying it as the primary bottleneck for current technology. While systems excel at "Exact Match" scenarios (linking identical strings), they struggle when semantic or discourse knowledge is required (e.g., linking "the charges" to "the sale of stock").

Conclusion

This diagnostic tool provides a path forward for coreference resolution research. The findings suggest that future efforts should shift focus from refining mention detection toward improving discourse and semantic modeling to address the persistent challenges of missing and divided entities. The tool is open-source and available for community use at code.google.com/p/berkeley-coreference-analyser/.

Summary updated

Abstract

Coreference resolution metrics quantify errors but do not analyze them. Here, we consider an automated method of categorizing errors in the output of a coreference system into intuitive underlying error types. Using this tool, we first compare the error distributions across a large set of systems, then analyze common errors across the top ten systems, empirically characterizing the major unsolved challenges of the coreference resolution task.
BibTeX
@inproceedings{kummerfeld2013error,
  author = {Jonathan K. Kummerfeld and Dan Klein},
  title = {{Error-Driven Analysis of Challenges in Coreference Resolution}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=SkEX2Wzu-B},
}
Figure