On Coreference Resolution Performance Metrics

Xiaoqiang LuoEMNLP 2005

TL;DR

This paper introduces a new metric, Constrained Entity-Alignment F-Measure (CEAF), for evaluating coreference resolution systems. CEAF addresses critical flaws in existing metrics like MUC and ACE-Value by ensuring a one-to-one mapping between system and reference entities, leading to more interpretable and discriminative performance scores.

What Problem Does This Paper Address?

Coreference resolution is the task of identifying all mentions in a text that refer to the same real-world entity. For example, in "The American Medical Association voted yesterday to install the heir apparent as its president-elect...", "The American Medical Association," "its," and "group" all refer to the same organization. Evaluating how well machines perform this task is crucial, but existing metrics have significant shortcomings. The widely-used MUC F-measure, for instance, can be misleading by favoring systems that produce fewer entities and struggles with single-mention entities. The ACE-Value, while better, is difficult to interpret, making it hard to understand what a given score truly means in terms of system performance. This paper aims to develop a metric that is both discriminative (clearly distinguishing good systems from bad ones) and interpretable (providing an intuitive understanding of performance).

Methods

The core contribution is the Constrained Entity-Alignment F-Measure (CEAF). CEAF works by finding the optimal one-to-one mapping between reference entities and system-generated entities. This alignment is framed as a maximum bipartite matching problem, which is efficiently solved using the Kuhn-Munkres algorithm.

The CEAF metric is defined using precision, recall, and F-measure, calculated based on an entity similarity measure. The paper proposes two such similarity measures:

  • Mention-based CEAF ($3(R, S)$): This measures similarity based on the number of common mentions between two entities. The F-measure then reflects the percentage of mentions that are correctly assigned to entities.
  • Entity-based CEAF ($4(R, S)$): This uses a relative measure (a mention F-measure between entities) to quantify similarity. The F-measure here can be interpreted as the ratio of correctly identified entities.

The key innovation is the constrained alignment: each reference entity can be matched to at most one system entity, and vice versa. This prevents systems from getting undue credit by reusing entities, a problem that plagues older metrics.

Datasets

The paper evaluates its proposed CEAF metric on:

  • Artificial data: Illustrated with examples in Figure 1 to highlight the differences between metrics.
  • MUC6 coreference test set: Used for comparing CEAF with the MUC F-measure.
  • ACE 2002 and 2004 data: Used for comparing CEAF with the ACE-Value.

The paper does not introduce new datasets but uses existing benchmarks to demonstrate the superiority of CEAF.

Key Insights and Ideas

  • The "one-to-one" constraint is critical: Older metrics often allow entities to be "intersected" multiple times, leading to inflated scores and counter-intuitive results. CEAF's strict one-to-one mapping ensures that each entity is considered only once, providing a more accurate reflection of performance.
  • Interpretability matters: The paper emphasizes that a good metric should be easy to understand. CEAF, by directly relating to the proportion of correct entities or mentions, offers a much clearer picture than the cost-based ACE-Value.
  • MUC's flaw: The MUC F-measure is shown to be highly susceptible to favoring systems that produce fewer entities, even if those entities are poorly formed. This lack of discriminativity is a significant drawback.
  • Symmetry is desirable: CEAF is noted to be a symmetric metric, meaning the order of comparison doesn't matter, which is a valuable property for evaluation.

Key Results

  • Artificial Data: On the example in Figure 1, CEAF correctly ranks system responses from best to worst, while MUC fails to distinguish between two responses, and B-cube produces counter-intuitive recall/precision scores.
  • MUC6 Data: When varying a penalty parameter, the mention-based CEAF shows a clear peak performance, indicating a system that produces a number of entities close to the ground truth. In contrast, the MUC F-measure monotonically increases as the system proposes fewer entities, highlighting its flaw.
  • ACE Data: CEAF's optimal performance is achieved with a number of proposed entities closer to the ground truth compared to the optimal ACE-Value system, suggesting CEAF better captures the "truth" of the system's output.

Limitations and Future Work

  • Similarity Measure Choice: While the paper proposes two reasonable entity similarity measures ($3$ and $4$), it acknowledges that other choices are possible. Future work could explore incorporating entity types or attributes into these similarity measures.
  • Computational Complexity: While the Kuhn-Munkres algorithm provides a polynomial-time solution, its complexity can still be a factor for very large documents. The paper mentions that skipping zero-similarity pairs can speed up computation.
  • Generalizability: The paper suggests that the CEAF paradigm can be applied to other tasks that involve comparing sets of objects, such as relation extraction, indicating a broader applicability beyond coreference resolution.

Summary updated

BibTeX
@inproceedings{luo2005coreference,
  author = {Xiaoqiang Luo},
  title = {{On Coreference Resolution Performance Metrics}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2005},
}
Figure