TL;DR
This paper critically examines the common metrics used to evaluate coreference resolution systems, arguing that current methods often overlook the "informational value" of mentions. By analyzing system outputs on real texts and comparing them to human judgment, the authors suggest that a more nuanced evaluation, considering the semantic weight of mentions, could lead to better system development and evaluation.
What Problem Does This Paper Address?
Coreference resolution (CR) is a fundamental task in natural language understanding, aiming to identify all expressions in a text that refer to the same real-world entity. While CR systems have seen significant progress, especially with shared tasks like CoNLL, the evaluation of these systems remains a challenge. Researchers often rely on a suite of metrics (MUC, B³, CEAF, BLANC) that primarily focus on abstract entity matching. However, the authors argue that these metrics may not fully capture how humans perceive the success or failure of CR. The problem is that current metrics might not adequately account for the loss or preservation of meaningful information within mentions, leading to a potential disconnect between automatic scores and human intuition about a system's true performance.
Methods
The paper revisits five commonly used coreference resolution evaluation metrics: MUC, B³, CEAF (two versions: CEAFe and CEAFm), and BLANC. These metrics are typically framed in terms of precision, recall, and F-score, comparing "key entities" (gold standard) to "response entities" (system output).
The authors then conduct an empirical analysis by applying these metrics to the output of the Stanford Multi-Pass Sieve Coreference Resolution System on five short texts from the OntoNotes corpus, representing different genres. Crucially, they go beyond just reporting metric scores. They categorize the key/response entity pairs into four groups based on a human-like assessment of informational value:
- Perfect match: Key and response entities are identical.
- Partial match: Response entities are subsets of key entities (further divided into partial entities and partial mentions).
- Merged entities: Response entities combine mentions from multiple key entities.
- Unsuccessful coreference resolution: Response entities are empty or contain incorrect/meaningless mentions.
They also analyze the correlation between different metrics using Kendall's τ rank correlation coefficient on results from CoNLL shared tasks.
Datasets
The paper uses five short texts from the OntoNotes 4.0 corpus. These texts are drawn from five different genres: Broadcast Conversations (BC), Broadcast News (BN), Magazine (MZ), News Wire (NW), and Web Blogs and News Groups (WB). The gold standard annotations for these texts contain 64 mentions distributed among 21 key entities. The system used for evaluation is the Stanford's Multi-Pass Sieve Coreference Resolution System.
Key Insights and Ideas
- Informational Value Matters: The core argument is that current metrics often fail to capture the "informational value" of mentions. A resolution might be considered "correct" by a metric even if it loses significant semantic content, and vice-versa.
- Human Intuition vs. Metrics: The authors propose that human judgment of CR success is more nuanced, considering not just entity matching but also the semantic weight and correctness of the information conveyed by mentions.
- "Fuzziness" in Evaluation: The analysis of correlations between metrics on CoNLL shared tasks reveals a degree of "fuzziness" and variation, suggesting that different metrics can provide significantly different assessments of the same system.
- Mention Detection Stringency: Stricter mention detection requirements (e.g., exact boundary matching) don't necessarily lead to better evaluation quality, as demonstrated by cases where partial matches with strict mention detection received lower scores despite comparable informational loss.
- "Full-bodied" vs. "Meaning-lean" Mentions: The semantic richness of mentions plays a role in human evaluation. A correct resolution involving a "full-bodied" noun phrase is perceived as more successful than one involving a "meaning-lean" pronoun, even if the metric scores are similar.
Key Results
- Low Metric Correlation: Correlations between metrics on CoNLL 2011 shared tasks were found to be relatively weak (as low as 0.22), though they improved in CoNLL 2012.
- Category Distribution: In their sample, "Perfect match" entities were rare (19% of entities, 14% of mentions), often achieved by simple string matching.
- Partial Match Nuances: Category 2a (partial entities) generally received higher scores than Category 2b (partial mentions), even when informational loss was comparable, highlighting the impact of strict mention detection.
- Merged Entities: Scores for merged entities (Category 3) were lower than partial matches but better than Category 2b.
- Unsuccessful Resolution Discrepancies: For Category 4 (unsuccessful resolution), link-based metrics (MUC, BLANC) ranked entities last, aligning with human intuition, while mention-based metrics (B³, CEAF) ranked them higher, indicating a significant disagreement.
- Entity MZ33 Example: This entity received high F-scores from all metrics, yet the response entity lost substantial information about the person's identity and role, illustrating the metrics' limitations.
Limitations and Future Work
- Manual Analysis: The authors acknowledge that their manual error analysis, while insightful, is limited by the small sample size and may not be fully representative.
- Metric Design: The metrics were primarily designed for evaluating whole texts, not individual key/response pairs, which introduces an approximation in their analysis.
- Future Work:
- Investigate the distribution of error categories on a larger sample.
- Formalize the proposed classification based on informational value.
- Incorporate more human evaluators for a more representative judgment.
- Extend evaluation from individual entity pairs to whole texts.
- Explore how to incorporate the "informational value" of mentions into both coreference resolution systems and their evaluation.
Summary updated