TL;DR
This paper introduces a novel evaluation scheme for "partial event coreference," a nuanced form of event relation detection that goes beyond simple identity. They propose a conceptual event hierarchy and adapt existing metrics (MUC, BLANC) to better assess systems that identify these complex event structures.
What Problem Does This Paper Address?
Event coreference resolution, the task of identifying when different mentions refer to the same event, is crucial for many NLP tasks. However, events aren't always identical; they can have sub-event or membership relationships, forming hierarchical structures. Existing evaluation metrics, designed for full coreference, struggle to capture these partial relationships, especially their directed nature and potential for complex hierarchies. This paper tackles the challenge of creating a robust and informative evaluation method for these partial event coreference systems, which is essential for advancing research in this area.
Methods
The authors propose an evaluation scheme based on a conceptual event hierarchy. Instead of directly evaluating event mentions, they abstract them into conceptual events, simplifying the representation of hierarchical structures. This hierarchy is built from unordered trees, reflecting the nature of partial coreference.
They then adapt three existing coreference evaluation metrics for this new scheme:
- MUC (Vilain et al., 1995): Extended to MUCp, this metric focuses on the minimum number of links needed to map system output to the gold standard, adapted to handle directed partial coreference links.
- BLANC (Recasens and Hovy, 2011): Extended to BLANCp, this metric, which computes precision, recall, and F1 scores for both coreference and non-coreference links, is also adapted for directed partial coreference.
- Simple Tree Matching (STM) (Yang, 1991): Extended to STMp, this metric, originally for ordered trees, is adapted for unordered trees using a greedy search approach.
The paper also defines desiderata for an ideal metric, including identity, symmetry, zero score for different forests, monotonicity, and linearity.
Datasets
The paper uses artificial data for their experiments, generated to represent a gold standard with fifty events, including singleton events and event trees with three levels or less. They also create system responses that range from all singletons to a single large tree merging all events. The authors note that this artificial distribution is intended to be representative of real-world data.
Key Insights and Ideas
- Conceptual Event Hierarchy: Abstracting event mentions into conceptual events is key to simplifying the evaluation of complex, hierarchical event structures. This approach effectively handles the "link propagation" problem where full coreference can imply partial coreference.
- Unordered Trees for Partial Coreference: Partial coreference naturally forms unordered trees, which requires metrics to be adapted from their original ordered tree or cluster-based formulations.
- MUC and BLANC Adaptability: Existing link-based metrics like MUC and BLANC are more readily adaptable to partial coreference than cluster-based metrics (like B-CUBED and CEAF), which struggle to represent hierarchical relationships.
- Desiderata for Evaluation: Defining clear desiderata (identity, monotonicity, linearity, etc.) is crucial for ensuring an evaluation metric is both informative and useful for tracking progress.
Key Results
The experimental results indicate that the extended MUC (MUCp) and BLANC (BLANCp) metrics are more adequate than the extended Simple Tree Matching (NSTMp) for evaluating partial event coreference. MUCp and BLANCp curves closely follow linearity, making them practically useful for tracking performance. NSTMp, on the other hand, fails to meet key desiderata like monotonicity and linearity, particularly when dealing with incorrect links or when systems produce a single large tree.
Limitations and Future Work
The authors acknowledge that while MUCp and BLANCp are promising, MUC has known disadvantages: it favors systems with more mentions per entity and ignores recall for singletons. BLANCp, while better, still requires the assumption of no twinless mentions.
Future work could involve:
- Incorporating structural consistency of event trees as an additional property for metrics.
- Implementing metrics from the perspective of broad contexts beyond local link-based counting.
- Exploring the flexibility of metrics by evaluating them without the assumptions of twinless mentions and intransitivity.
Summary updated