Open-Domain Event Detection using Distant Supervision

Jun Araki, Teruko MitamuraOpenReview 2018 · OpenReview

Abstract

Traditional event detection systems are often limited to narrow, predefined domains due to the high cost of human annotation. This paper presents a novel framework for open-domain event detection that leverages distant supervision from WordNet and Wikipedia. By automatically generating high-quality training data, the proposed system identifies a wide range of event types across diverse genres. Experimental results demonstrate that this approach outperforms traditional supervised models, achieving a 71.3% Strict F1 score on a new multi-domain benchmark.

Introduction: The Challenge of Open-Domain Detection

Most existing event detection research focuses on "closed-domain" scenarios, where models are trained on specific categories (e.g., "Conflict" or "Justice") within restricted genres like newswire. While effective in their niche, these models struggle with the "open-domain" reality of the web, where events can range from geological shifts to economic fluctuations.

The primary obstacle to scaling these systems is the "human-in-the-loop" bottleneck. Manually labeling every possible event type across all domains is prohibitively expensive. To address this, the authors explore whether "event-ness" can be learned as a general linguistic property through distant supervision, eliminating the need for exhaustive human labeling.

Methodology

The authors propose a two-phase pipeline: Training Data Generation and Event Detection.

Overview of the distantly-supervised open-domain event detection pipeline.

Phase 1: Training Data Generation

The system uses WordNet to identify "eventualities"—actions, processes, and states. By mapping WordNet's taxonomic hierarchy to these categories, the authors automatically label verbs and nouns in the SemCor corpus.

To extend coverage to proper nouns and real-world events not found in standard dictionaries (e.g., specific historical events), the authors utilize Wikipedia. They train a specialized neural classifier, GC-BLSTM-Attn, on WordNet's textual definitions ("glosses"). This classifier learns to distinguish "eventive" definitions (e.g., "the act of paying money") from non-eventive ones (e.g., "meat from a domestic hog"). This classifier is then applied to Wikipedia article introductions to harvest new training examples.

Phase 2: Event Detection Model

The core detection model is a Bidirectional LSTM (BLSTM) equipped with a self-attention mechanism.

The BiLSTM model architecture with self-attention used for gloss classification and event detection.

The model treats event detection as a sequence labeling task. To handle complex linguistic structures like multi-word expressions or discontinuous phrases (e.g., "turned the TV on"), the authors employ an extended tagging scheme (BIO + DB/DI). This allows the model to capture event "nuggets" that span multiple non-contiguous words.

Evaluation and Results

The system was evaluated against a new gold-standard benchmark, SW100, consisting of 100 articles from Simple English Wikipedia across 10 diverse domains (e.g., Architecture, Chemistry, Disaster, Economics).

Key results include:

  • Superior Performance: The distantly supervised model (DS-BLSTM) achieved a 71.3% Strict F1 score, outperforming a standard supervised BLSTM (67.9%) trained on limited human-labeled data.
  • Gloss Classification Accuracy: The attention-based gloss classifier reached 91.65% accuracy on WordNet and 85.00% on unseen Wikipedia concepts.
  • Cross-Domain Robustness: The model maintained consistent performance across all 10 domains, suggesting that the learned features of "event-ness" are largely domain-independent.

Discussion: Insights and Limitations

A significant finding is the utility of dictionary "glosses" as a bridge between structured knowledge bases and unstructured text. The linguistic patterns used to define events are remarkably stable, allowing a model trained on definitions to generalize to real-world mentions. Furthermore, the study suggests that the sheer volume of distantly supervised data (11 times larger than typical human-labeled sets) compensates for the inherent noise in automatic labeling.

However, limitations remain. The model occasionally struggles with ambiguous nouns—for instance, distinguishing "payment" as an action from "payment" as a physical sum of money. Additionally, highly technical or rare domain-specific terms may be missed if they lack sufficient representation in the underlying knowledge sources.

Conclusion

This work demonstrates that distant supervision is a viable and powerful alternative to manual annotation for open-domain event detection. By treating "event-ness" as a universal linguistic property and leveraging the vast knowledge in WordNet and Wikipedia, the authors provide a scalable path forward for understanding events across the entire web.

Summary updated

Abstract

This paper introduces open-domain event detection, a new event detection paradigm to address issues of prior work on restricted domains and event annotation. The goal is to detect all kinds of events regardless of domains. Given the absence of training data, we propose a distant supervision method that is able to generate high-quality training data. Using a manually annotated event corpus as gold standard, our experiments show that despite no direct supervision, the model outperforms supervised models. This result indicates that the distant supervision enables robust event detection in various domains, while obviating the need for human annotation of events.
BibTeX
@misc{araki2018open,
  author = {Jun Araki and Teruko Mitamura},
  title = {{Open-Domain Event Detection using Distant Supervision}},
  year = {2018},
  url = {https://openreview.net/forum?id=JZfVYWQZ_N},
  howpublished = {OpenReview},
}
Figure