Automatic Event Extraction with Structured Preference Modeling

Wei Lu, Dan RothACL 2012 · OpenReview

Abstract

This paper presents a novel sequence labeling model based on the latent-variable semi-Markov conditional random fields for jointly extracting argument roles of events from texts. The model takes in coarse mention and type information and predicts argument roles for a given event template. Addressing the event extraction problem in a primarily unsupervised setting, the authors introduce a novel learning framework called structured preference modeling (PM). This framework allows arbitrary preferences to be assigned to certain structures during the learning procedure. Empirically, structured preferences are shown to be crucial to the success of the task, yielding performance competitive with supervised approaches despite the lack of annotated training data.

Introduction

Event extraction—the task of identifying "who did what to whom, when, and where"—is a cornerstone of structured information retrieval. Traditionally, this task relies on massive human-annotated datasets and near-perfect entity detection, both of which are difficult to obtain and maintain across domains. This paper introduces a novel unsupervised framework that replaces the need for labeled data with a small set of "structured preferences." By leveraging a latent-variable semi-Markov Conditional Random Field (CRF) guided by these human-defined preferences, the model achieves performance competitive with supervised baselines while remaining robust to the noise inherent in real-world text.

Methodology

The proposed system introduces two major technical innovations designed to handle the complexities of unsupervised event extraction.

Latent-Variable Semi-Markov CRF

Unlike standard sequence models that process text word-by-word, a semi-Markov CRF operates on segments or spans of words. This is essential for event extraction because arguments (e.g., "North Korea's military") are often multi-word phrases. The model distinguishes between Argument Segments, which fill roles in an event template, and Gap Segments, which consist of the words between arguments. These "gaps" are highly informative, as they typically contain the verbs and prepositions that define the relationships between entities. Because the model is unsupervised, it treats the "true" mention types as latent variables that must be inferred during training.

Structured Preference Modeling (PM)

The core contribution is the Structured Preference Modeling (PM) framework. Instead of learning from labeled examples, the model is guided by "soft" preferences. Unlike hard constraints, preferences reward structures that align with simple, human-defined patterns.

Examples of structured preference patterns used to guide the unsupervised learning process for different event types.

For instance, a preference might suggest that a TIME argument often follows the word "in". The learning objective optimizes a function that shifts probability mass toward these preferred structures, allowing the model to generalize from a few simple rules to the complex features of the entire dataset.

Experimental Setup

The model was evaluated using the newswire portion of the ACE 2005 (Automatic Content Extraction) dataset. The study focused on the four most frequent event types: Attack, Meet, Die, and Transport. To test robustness, the authors evaluated the system using both "gold" (perfect) mention boundaries and "automatic" (noisy) boundaries generated by an automated system.

Results and Discussion

The Structured Preference Modeling (PM) approach significantly outperformed purely unsupervised and rule-based baselines.

Performance comparison (F1-score) of the PM approach against unsupervised and supervised baselines across different event types.

Key findings include:

  • Performance Gains: For the "Attack" event, the PM model achieved an F1-score of 42.02%, a substantial improvement over the 30.12% achieved by a standard unsupervised model.
  • Supervised Competitiveness: In several instances, the unsupervised PM model performed comparably to or better than supervised Maximum Entropy (MaxEnt) baselines trained on labeled data.
  • Robustness to Noise: The PM model maintained respectable performance even with noisy, automatically predicted mention tags (e.g., 49.73% F1 for "Transport"), demonstrating its utility in real-world NLP pipelines.
  • Efficiency of Knowledge: The system required only 6 to 7 simple preference patterns per event type to anchor the learning process effectively.

Conclusion

This work demonstrates that "a little knowledge goes a long way" in unsupervised event extraction. By using structured preferences rather than hard constraints, the model remains flexible enough to handle the variability of natural language. While the current approach requires manual pattern engineering and assumes non-overlapping arguments, it provides a powerful framework for scaling event extraction to new domains without the prohibitive cost of manual annotation. Future research may focus on automatic pattern discovery and extending the model to support overlapping argument structures.

Summary updated

Abstract

This paper presents a novel sequence labeling model based on the latent-variable semi-Markov conditional random fields for jointly extracting argument roles of events from texts. The model takes in coarse mention and type information and predicts argument roles for a given event template. This paper addresses the event extraction problem in a primarily unsupervised setting, where no labeled training instances are available. Our key contribution is a novel learning framework called structured preference modeling (PM), that allows arbitrary preference to be assigned to certain structures during the learning procedure. We establish and discuss connections between this framework and other existing works. We show empirically that the structured preferences are crucial to the success of our task. Our model, trained without annotated data and with a small number of structured preferences, yields performance competitive to some baseline supervised approaches.
BibTeX
@inproceedings{lu2012automatic,
  author = {Wei Lu and Dan Roth},
  title = {{Automatic Event Extraction with Structured Preference Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2012},
  url = {https://openreview.net/forum?id=H1WEGaxuZr},
}
Figure