New Insights into Cross-Document Event Coreference: Systematic Comparison and a Simplified Approach

Andres Cremisini, Mark FinlaysonACL 2020

TL;DR

This paper tackles the challenge of identifying when events described in different documents refer to the same real-world occurrence. They introduce a simplified, event-trigger-only approach that matches state-of-the-art performance on the Event Coreference Bank+ (ECB+) dataset, while being more interpretable and requiring fewer resources. This work also provides a crucial cross-validated evaluation of the ECB+ dataset, offering a clearer picture of existing methods and best practices.

What Problem Does This Paper Address?

Cross-Document Event Coreference (CDEC) is the task of linking event mentions across multiple documents that refer to the same real-world event. Imagine reading news articles about a specific protest: CDEC aims to identify all mentions of that protest, even if they appear in different articles, written by different authors, and published on different days. This is a complex problem because events can be described in many ways, with varying levels of detail and different surrounding contexts.

Existing CDEC systems often rely on both event triggers (the core verb or noun describing the event) and entities (participants, locations, times) to establish coreference. However, comparing these systems is difficult due to variations in how they use annotated data (gold-standard vs. automatically extracted) and how they preprocess documents into topical clusters. This paper aims to simplify the CDEC process by focusing solely on event triggers and providing a more standardized and transparent evaluation framework.

Methods

The authors build upon the "BAG OF EVENTS" system, known for its simplicity and strong performance. Their approach involves two main stages:

  1. Document Clustering: Documents are first grouped into topical clusters. This is done using an affinity propagation algorithm on bag-of-words representations with tf-idf weights. This step significantly reduces the number of event pairs to consider, making the subsequent event clustering more tractable.
  2. Pairwise Event Coreference Classifier: A shallow, fully-connected neural network with one hidden layer is trained to predict whether two events within the same document cluster are coreferent. This classifier uses a significantly reduced feature set (4 features) compared to prior work (which used up to 19). The key features include:
  3. Head Phrase Word Similarity (Vec): Measures the semantic similarity of event triggers using pre-trained word embeddings. This is found to be the most impactful feature.
  4. Event Word Distribution (WD): Captures lexical similarity of the event's textual expression.
  5. Relative Sentence Similarity (SS): Captures the lexical similarity of the sentences in which the events appear.
  6. Head Phrase Part of Speech (POS): A binary feature indicating if event triggers share the same part of speech.

The event clustering then uses the confidence scores from this pairwise classifier to form coreference chains.

Datasets

The primary dataset used for training and evaluation is the Event Coreference Bank+ (ECB+) corpus. The authors specifically use a subset of 1,840 sentences that have been double-checked for coreference annotations. This subset contains 5,726 events and 897 coreference chains. The ECB+ corpus is an extension of the Event Coreference Bank (ECB) and is designed for cross-document event coreference tasks.

Key Insights and Ideas

  • Simplicity Can Be Powerful: The paper demonstrates that a simplified approach focusing only on event triggers, with a reduced feature set, can achieve state-of-the-art performance. This challenges the notion that complex systems incorporating entities are always necessary.
  • Feature Importance: The "Head Phrase Word Similarity" feature, leveraging pre-trained word embeddings, is identified as the most crucial component for accurate pairwise event coreference prediction.
  • Document Clustering's Impact: While document clustering is a vital pre-processing step that dramatically reduces the search space and improves efficiency, its direct contribution to CDEC performance (measured by CONLL F1) is modest (around 3 points). This suggests that while necessary for tractability, improvements in document clustering alone may not lead to massive gains in CDEC accuracy.
  • Need for Standardized Evaluation: The paper highlights the significant challenges in comparing prior CDEC systems due to variations in testing schemes and annotation usage. Their introduction of a cross-validated evaluation on ECB+ is a significant step towards more reproducible and comparable research.

Key Results

  • The proposed simplified system achieves comparable state-of-the-art performance on the ECB+ dataset, with a 5-fold cross-validated CONLL F1 score of 71.1.
  • The "Head Phrase Word Similarity" feature alone, when used with logistic regression, achieves a CONLL F1 of 70.7 (and 69.2 on a specific test set), underscoring its importance.
  • Document clustering reduces the false/true event pair ratio from 89:1 to 6:1, making the problem tractable, though it results in a loss of about 13% of true coreferring event pairs.
  • The pairwise event coreference classifier, in isolation, achieves a maximum F1 score of 0.59 with a cutoff of 0.72.

Limitations and Future Work

  • Event Trigger Detection: The paper acknowledges that the performance of event trigger detection is a major bottleneck. Their system uses a pre-trained system (CAEVO), which achieved modest performance on ECB+. Future work could explore more advanced event extraction techniques.
  • Document Clustering Generalizability: The authors note that the strong performance of their document clustering on ECB+ might not generalize to "documents in the wild," as ECB+ clusters are often quite distinct and lexically separable.
  • Cross-Validation for Comparison: While the paper provides cross-validated results, it also points out the difficulty in directly comparing with all prior work due to differing evaluation methodologies. They advocate for future CDEC systems to report performance with and without gold-standard document clusters.
  • Future Directions: The authors suggest that future work could explore more sophisticated event clustering techniques beyond simple transitive closure based on confidence thresholds. They also implicitly suggest that further investigation into the impact of different feature sets and classifier architectures could be beneficial.

Summary updated

BibTeX
@inproceedings{cremisini2020new,
  author = {Andres Cremisini and Mark Finlayson},
  title = {{New Insights into Cross-Document Event Coreference: Systematic Comparison and a Simplified Approach}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure