DEGREE: A Data-Efficient Generative Event Extraction Model

I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, S.l. Miller, Prem Natarajan, Kai-Wei Chang, Nanyun Peng

Abstract

Event extraction (EE) aims to identify structured events, including event triggers and their corresponding arguments, from unstructured text. Most of the existing works rely on a large number of labeled instances to train models, while the labeled data could be expensive to be obtained. In this work, we present a data-efficient event extraction method by formulating event extraction as a natural language generation problem. The formulation allows us to inject knowledge of label semantics, event structure, and output dependencies into the model. Given a passage and an event type, our model learns to summarize this passage into a templated sentence in a predefined structure. The template is event-type-specific, manually created, and contains event trigger and argument information. Lastly, a rule-based algorithm is used to derive the trigger and argument predictions from the generated sentence. Our method inherently enjoys the following benefits: (1) The pretraining of the generative language models help incorporate the semantics of the labels for generative EE. (2) The autoregressive generation process and our end-to-end design for extracting triggers and arguments force the model to capture the dependencies among the output triggers and their arguments. (3) The predefined templates form concrete yet flexible rules to hint the models about the valid patterns for each event type, reducing the models' burden to learn structures from the data. Empirical results show that our model achieves superior performance over strong baselines on EE tasks in the low data regime and achieves competitive results to the current state-of-the-art when more data becomes available.

Introduction

Event Extraction (EE) is the task of identifying "what happened" in a text by finding triggers (words signaling an event, like "execution") and arguments (participants and their roles, like the "Agent" or "Place"). Traditionally, this is treated as a multi-stage classification problem. However, this approach suffers from high data dependency and a failure to capture dependencies between subtasks.

DEGREE (Data-Efficient GeneRation-Based Event Extraction) addresses these flaws by transforming EE into a conditional generation problem. By using manually designed natural language prompts and templates, it leverages the semantic knowledge of pre-trained language models to achieve state-of-the-art performance in low-resource settings where labeled data is scarce.

Methodology

The core innovation of DEGREE is formulating EE as conditional natural language generation. Instead of predicting labels, the model (based on BART-large) takes a passage and a structured prompt as input and is trained to "summarize" the event into a predefined natural language template.

The DEGREE framework: A passage and event-specific prompt are fed into an encoder-decoder model to generate a filled template.

The Prompt Components

  • Event Type Definition: A short description of the event (e.g., "The event is related to conflict...").
  • Event Keywords: Words typically associated with the event (e.g., "war, attack, terrorism").
  • E2E Template: A sentence structure with placeholders. For a "Conflict:Attack" event, the template might be: Event trigger is <Trigger>. <some attacker> attacked <some facility>...

The Workflow

  1. Input: The model receives the passage and the prompt.
  2. Generation: The model fills in the template placeholders using information from the passage. For example, it might generate: "Event trigger is detonated. Palestinian attacked jeep..."
  3. Inference: A deterministic algorithm maps the filled-in words back to the original text spans to provide the final structured prediction.

Experimental Results

The researchers evaluated DEGREE on standard benchmarks including ACE 2005 and ERE-EN. To test data efficiency, they created "low-resource" splits, training the model on as little as 1%, 3%, and 5% of the total available data.

Performance comparison of DEGREE against baselines across varying percentages of training data on ACE05 and ERE-EN datasets.

DEGREE shines brightest when data is most limited. On the ACE05-E dataset with only 1% of training data, DEGREE achieved a Trigger F1 score of 55.4, outperforming the strong OneIE baseline by over 15 points. In the same 1% setting, DEGREE's Argument F1 score was 21.7, more than double that of OneIE (9.4). Even when 100% of the data is available, DEGREE remains competitive with or superior to state-of-the-art classification models.

Key Insights

  • Label Semantics as a Guide: Using words like "somewhere" or "someone" in templates provides semantic hints about the type of entity required (e.g., location vs. person).
  • Natural Language Advantage: Natural sentences in templates outperform special tokens because they allow the model to tap into the linguistic knowledge acquired during pre-training.
  • Joint Learning: Predicting triggers and arguments simultaneously forces the model to learn structural dependencies, which is critical in low-resource scenarios.

Conclusion and Future Work

While DEGREE demonstrates exceptional data efficiency, it is sensitive to template wording, and inference can be computationally expensive when iterating through many event types. Future work will focus on automating template generation from existing ontologies to further reduce human effort and improve scalability.

Summary updated

Abstract

Event extraction (EE) aims to identify structured events, including event triggers and their corresponding arguments, from unstructured text. Most of the existing works rely on a large number of labeled instances to train models, while the labeled data could be expensive to be obtained. In this work, we present a data-efficient event extraction method by formulating event extraction as a natural language generation problem. The formulation allows us to inject knowledge of label semantics, event structure, and output dependencies into the model. Given a passage and an event type, our model learns to summarize this passage into a templated sentence in a predefined structure. The template is event-type-specific, manually created, and contains event trigger and argument information. Lastly, a rule-based algorithm is used to derive the trigger and argument predictions from the generated sentence. Our method inherently enjoys the following benefits: (1) The pretraining of the generative language models help incorporate the semantics of the labels for generative EE. (2) The autoregressive generation process and our end-to-end design for extracting triggers and arguments force the model to capture the dependencies among the output triggers and their arguments. (3) The predefined templates form concrete yet flexible rules to hint the models about the valid patterns for each event type, reducing the models' burden to learn structures from the data. Empirical results show that our model achieves superior performance over strong baselines on EE tasks in the low data regime and achieves competitive results to the current state-of-the-art when more data becomes available.
BibTeX
@inproceedings{hsu2022degree,
  author = {I-Hung Hsu and Kuan-Hao Huang and Elizabeth Boschee and S.L. Miller and Prem Natarajan and Kai-Wei Chang and Nanyun Peng},
  title = {{DEGREE: A Data-Efficient Generative Event Extraction Model}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.138},
}
Figure