CMU-LTI at KBP 2015 Event Track

Zhengzhong Liu, Jun Araki, Dheeru Dua, Teruko Mitamura, Eduard H. Hovy

Introduction

The Carnegie Mellon University Language Technologies Institute (CMU-LTI) participated in the KBP 2015 Event Track, a benchmark competition focused on extracting and linking events from unstructured text. The task requires identifying "event nuggets"—the specific lexical triggers of an event—and performing "event coreference" to determine if multiple mentions refer to the same real-world occurrence. The CMU-LTI approach combines structured learning techniques with a rich set of linguistic and distributional features to address the complexities of multi-labeled event mentions and semantic paraphrasing.

Methodology

Event Nugget Detection

The team developed two primary systems for nugget detection to explore different architectural trade-offs:

  1. LTI1: This system utilizes a Conditional Random Field (CRF) trained via a Structured Perceptron for joint span and type detection. Realis classification is handled by a subsequent linear Support Vector Machine (SVM). To address "double-tagging"—where a single word triggers multiple event types (e.g., "killed" indicating both Conflict.Attack and Life.Die)—LTI1 employs a label concatenation strategy, treating multi-label combinations as unique classes.
  2. LTI2: This variant also uses a CRF but introduces a k-Best Viterbi merging algorithm. By examining the top-5 most likely sequences, the system can extract overlapping event types for a single trigger word.

Event Coreference

For coreference, both systems utilize the Latent Antecedent Tree (LAT) model. This model represents coreference as a tree structure where each mention links to a latent previous mention. Training is conducted using a Passive-Aggressive (PA) algorithm, which proved essential for model stability and convergence.

The feature set for coreference includes:

  • Linguistic Features: POS tags, lemmas, and Semantic Role Labeling (SRL) derived from Fanse and Semafor parsers.
  • External Knowledge: WordNet hierarchies for synonymy and Brown clusters for reducing feature sparsity.
  • Distributional Semantics: 300-dimensional Word2Vec embeddings (trained on Google News) to calculate cosine similarity between triggers, enabling the system to link semantically related paraphrases.

Results and Evaluation

The systems were evaluated on the KBP 2015 Event Track dataset following RichERE guidelines.

Nugget Detection Performance

The LTI1 system demonstrated strong performance, ranking 2nd among all participants. It achieved an official F1 score of 41.77 for the combined Type and Realis task.

Official Nugget Detection results for the LTI1 system, showing performance across Span, Type, and Realis attributes.

Coreference and Stability

While LTI1 officially ranked 3rd in coreference, post-competition analysis revealed that fixing implementation bugs in the feature set significantly improved results. On the development set, the corrected system reached an F1 of 76.78. Cross-validation results further illustrate the model's performance stability across different data subsets.

Performance metrics across a 5-fold cross-validation, demonstrating consistency in the model's predictive capabilities.

Discussion and Key Insights

The CMU-LTI approach highlights several critical factors for successful event extraction:

  • Modeling Multi-labels: The "double tagging" intuition—that certain words naturally imply multiple event types—can be effectively captured through simple label concatenation.
  • Algorithmic Choice: The Passive-Aggressive algorithm was vital for training the Latent Antecedent Tree model effectively, ensuring convergence in a complex structured learning task.
  • Semantic Bridges: Word embeddings serve as a crucial bridge for coreference, allowing the system to move beyond simple string matching to capture semantic similarity between lexically different triggers.

Future work will explore joint modeling of detection and coreference to allow for mutual information exchange, as well as more automated methods for exploiting ontology hierarchies and external knowledge bases.

Summary updated

BibTeX
@inproceedings{liu2015cmu,
  author = {Zhengzhong Liu and Jun Araki and Dheeru Dua and Teruko Mitamura and Eduard H. Hovy},
  title = {{CMU-LTI at KBP 2015 Event Track}},
  booktitle = {TAC},
  year = {2015},
  url = {https://openreview.net/forum?id=HIagz7yJxb3},
}
Figure