EDM3: Event Detection as Multi-task Text Generation

Ujjwala Anantheswaran, Himanshu Gupta, Mihir Parmar, Kuntal Kumar Pal, Chitta Baral

Event Detection (ED) is a cornerstone of Natural Language Processing, tasked with identifying what happened in a text (the event trigger) and what kind of thing it was (the event type). Traditionally, this has been treated as a sequence labeling problem—essentially a fancy version of part-of-speech tagging. However, a new paper from researchers at Arizona State University suggests that we’ve been looking at it the wrong way.

In EDM3: Event Detection as Multi-task Text Generation, the authors propose shifting from discriminative token classification to a generative, text-to-text approach. By reframing ED as a multi-task generation problem guided by instructional prompts, they achieve state-of-the-art results on complex benchmarks.

TL;DR

EDM3 reframes event detection from a token classification task into a multi-task text generation challenge using a T5 model. By training the model to simultaneously identify, classify, and detect events using instructional prompts, the authors achieve state-of-the-art results on complex datasets and significantly improve handling of real-world nuances like multi-word and multi-class triggers.

What Problem Does This Paper Address?

Event Detection typically involves two subtasks: Event Identification (EI) (finding the trigger word, like "purchasing") and Event Classification (EC) (assigning a type, like Transaction).

Existing methods face three major hurdles:

  1. Error Propagation: Most systems are pipelined. If the identification step misses a trigger, the classification step never even gets a chance, leading to cascading failures.
  2. Rigidity: Traditional models are often tied to specific classification schemas or domains, making them hard to adapt to new types of data (e.g., moving from news to biomedical text).
  3. Complexity Mismatch: Real-world events aren't always single words. Triggers can be multi-word phrases ("took place") or even multi-class (one word triggering two different event types). Token classification models struggle to represent these complexities naturally.

Methods

The authors introduce EDM3 (Event Detection by Multi-task text generation over 3 subtasks). Instead of tagging tokens, they use a T5 (Text-to-Text Transfer Transformer) model to generate structured strings.

1. Generative Reformulation

The core idea is to convert labels into a delimited string format that the model learns to produce.

  • EI: Generates a list of triggers: ended | War
  • EC: Generates a list of types: process_end | military_operation
  • ED: Generates the combined mapping: ended->process_end | War->military_operation
Illustration of generative reformulation

2. Multi-task Learning

Rather than just training on the final ED task, the model is trained on all three tasks (EI, EC, and ED) simultaneously. This allows the model to learn "transferable knowledge"—learning how to identify triggers in isolation helps it perform the combined detection task better.

3. Instructional Prompt Tuning

To guide the generative process, the authors use natural language prompts that include a task definition, a general example, and even a domain-specific example (like a biomedical instance) before the actual input sentence.

Figure 2: An example of an input instance for reformu- lated generative ED. The input comprises a task defini- tion followed by diverse domain examples before the input sentence containing the events to be detected.

Datasets

The authors evaluated EDM3 across four diverse datasets to test its robustness and domain adaptability:

  • RAMS: Focused on multi-sentence context.
  • WikiEvents: Contains a high proportion of "negative" instances (sentences with no events).
  • MAVEN: A massive general-domain dataset with a long-tail distribution of 168 event types.
  • MLEE: A specialized biomedical domain dataset.
Table 2: Dataset statistics, including number of docu- ments per data split, as well as number of event triggers and unique event types across the dataset.

Key Insights and Ideas

  • Generative is more natural for complex triggers: Because the model generates text rather than tagging fixed tokens, it handles multi-word triggers (like "land grab") much better than traditional models. It achieved over 90% exact match accuracy on multi-word triggers in MAVEN.
  • Subtasks boost the main task: Training on EI and EC explicitly improved ED performance by at least 3% across all datasets compared to single-task training.
  • Prompt diversity is a "free" win: The authors found that including a biomedical example in the prompt for a general-domain task actually improved performance, suggesting that diverse examples help the model learn more generalized "event-ness."
  • The "Negative" Problem: Generative models struggle more than discriminative ones when a dataset has many sentences where nothing happens (negative instances). In WikiEvents, where 54% of sentences are negatives, the model's performance dropped significantly compared to its performance on only event-containing sentences.

Key Results

EDM3 demonstrated strong performance, particularly on datasets requiring complex reasoning:

  • State-of-the-Art on RAMS: Achieved a 71.3% F1 score, a significant jump over the previous SOTA of 65.1%.
  • Competitive on MLEE: Reached 78.1% F1 without using any domain-specific biomedical embeddings, outperforming several specialized models.
  • Overall Improvement: Outperformed single-task generation by an average of 8.4% and multi-tasking without prompts by 2.4%.
Results on RAMS

Limitations and Future Work

While EDM3 is powerful, the authors acknowledge a few key limitations:

  • Distinguishing Non-Events: The model faces difficulty in correctly identifying "NONE" for sentences without events. A dedicated binary classifier for event presence might be a necessary pre-processing step.
  • Contextual Limits: While it handles multi-sentence data better than many, it still struggles with very broad document-level context where saliency is determined by events far apart in the text.
  • Decoding Efficiency: The use of large beam searches (50 beams) for decoding is computationally expensive, suggesting a need for more efficient inference methods.

Future work involves exploring better prompt engineering and integrating external domain knowledge to further refine event classification.

Summary updated

Abstract

Event detection refers to identifying event occurrences in a text and comprises of two subtasks; event identification and classification. We present EDM3, a novel approach for Event Detection that formulates three generative tasks: identification, classification, and combined detection. We show that EDM3 helps to learn transferable knowledge that can be leveraged to perform Event Detection and its subtasks concurrently, mitigating the error propagation inherent in pipelined approaches. Unlike previous dataset- or domain-specific approaches, EDM3 utilizes the existing knowledge of language models, allowing it to be trained over any classification schema. We evaluate EDM3 on multiple event detection datasets: RAMS, WikiEvents, MAVEN, and MLEE, showing that EDM3 outperforms 1) single-task performance by 8.4% on average and 2) multi-task performance without instructional prompts by 2.4% on average. We obtain SOTA results on RAMS (71.3% vs. 65.1% F-1) and competitive performance on other datasets. We analyze our approach to demonstrate its efficacy in low-resource and multi-sentence settings. We also show the effectiveness of this approach on non-standard event configurations such as multi-word and multi-class event triggers. Overall, our results show that EDM3 is a promising approach for Event Detection that has the potential for real-world applications.
BibTeX
@inproceedings{anantheswaran2024edm,
  author = {Ujjwala Anantheswaran and Himanshu Gupta and Mihir Parmar and Kuntal Kumar Pal and Chitta Baral},
  title = {{EDM3: Event Detection as Multi-task Text Generation}},
  booktitle = {SEM @ NAACL},
  year = {2024},
  doi = {10.18653/v1/2024.starsem-1.35},
}
Figure