OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding

Hao Peng, Xiaozhi Wang, Feng Yao, Zimu Wang, Chuzhao Zhu, Kaisheng Zeng, Lei Hou, Juanzi Li

Introduction

Event understanding is a pivotal sub-field of Natural Language Processing (NLP) that seeks to transform unstructured text into structured event knowledge. This process typically involves three distinct stages: Event Detection (ED), which identifies event triggers; Event Argument Extraction (EAE), which determines the participants and attributes of those events; and Event Relation Extraction (ERE), which maps the relationships—such as temporal, causal, or coreference—between events.

Despite its importance, the field has historically been plagued by fragmentation. Most research focuses on ED and EAE in isolation, often neglecting the complex ERE phase. Furthermore, a lack of standardized evaluation protocols has led to "evaluation pitfalls," where differences in preprocessing scripts and output formats make direct model comparisons nearly impossible. To address these challenges, we introduce OmniEvent, an open-source toolkit designed to provide a comprehensive, fair, and efficient framework for the entire event understanding pipeline.

The Challenges of Event Understanding

Historically, existing NLP systems have treated event extraction tasks as isolated problems. While ED and EAE are well-studied, ERE remains a bottleneck due to its complexity. Beyond task isolation, the field suffers from severe inconsistencies in benchmarking. Because different researchers use varying data preprocessing scripts and pipeline evaluation methods, comparing two models often results in an "apples-to-oranges" comparison. Finally, existing toolkits often lack native, efficient support for integrating and fine-tuning modern Large Language Models (LLMs), which have become the backbone of state-of-the-art NLP.

The OmniEvent Architecture: Modularity and Paradigms

OmniEvent is built upon a modular architecture that allows researchers to disassemble and reconfigure mainstream models with ease. The toolkit decomposes models into three interchangeable components:

  1. Backbone: Supports a wide range of encoders, from traditional CNNs and LSTMs to modern Transformers like BERT and Large Language Models such as T5 and FLAN-UL2.
  2. Aggregation: Provides various layers for feature representation, including pooling mechanisms and Graph Neural Networks (GNNs).
  3. Classification Head: Offers diverse output layers, such as Conditional Random Fields (CRFs), multi-class classifiers, and antecedent ranking for coreference.

This modularity enables researchers to swap components—for instance, replacing a BERT backbone with a GPT-2 model—with minimal code changes. To accommodate the variety of modeling approaches in the literature, OmniEvent supports four distinct modeling paradigms: classification, sequence labeling, span prediction, and conditional generation.

Standardizing Evaluation and Data

One of the primary contributions of OmniEvent is its commitment to fair benchmarking. The toolkit standardizes 15 widely-used English and Chinese datasets into a unified JSON-based format. Supported datasets include ACE 2005, TAC KBP (2014-2017), RichERE, MAVEN, and MATRES, among others. By providing consistent preprocessing scripts, OmniEvent ensures that performance differences between models are due to architectural improvements rather than data manipulation.

Furthermore, OmniEvent introduces a unified evaluation protocol. It includes a conversion tool that maps the predictions of all four modeling paradigms into a single standardized output space. This allows for a truly fair comparison, particularly in pipeline evaluations where the quality of predicted triggers significantly impacts downstream argument extraction.

Large Language Model Integration

As the NLP field shifts toward Large Language Models, OmniEvent provides native support for fine-tuning and inference of models with billions of parameters. Built on the Hugging Face Transformers library and integrated with DeepSpeed, the toolkit facilitates efficient distributed training.

Our analysis reveals that while LLMs are exceptionally capable at extraction tasks (ED and EAE), they encounter significant difficulties with Event Relation Extraction (ERE). This is likely due to the complex output spaces and the long-range context required to construct event knowledge graphs. OmniEvent’s modularity allows researchers to focus on these specific weaknesses by easily integrating massive LLMs into specialized event extraction pipelines.

Benchmarking and Performance

OmniEvent successfully reproduces representative models across all supported paradigms, achieving results consistent with or superior to those reported in original papers. For example, on the ACE 2005 dataset, Event Detection F1 scores ranged between 66.0% and 71.0% depending on the paradigm used. The toolkit also establishes strong baselines for ERE, achieving an 87.7% F1 score for coreference on ACE 2005 and 90.4% F1 for temporal relations on MATRES.

Efficiency is a core feature of the toolkit. Using OmniEvent’s DeepSpeed integration, fine-tuning a 20-billion parameter model (FLAN-UL2) on the ACE 2005 dataset required only 25 GPU hours on NVIDIA A100s. This demonstrates that OmniEvent is not only a research tool but also a practical solution for modern, resource-intensive workflows.

Limitations and Future Directions

While OmniEvent represents a significant step forward, it currently focuses on sentence-level extraction. Future iterations aim to support document-level datasets, such as RAMS and WikiEvents, which require understanding context across multiple sentences. Additionally, the authors plan to expand language support—currently limited to English and Chinese—and integrate more advanced ERE techniques like Integer Linear Programming (ILP) inference.

By providing a unified, modular, and LLM-friendly environment, OmniEvent lowers the barrier to entry for event understanding research and paves the way for more robust and comparable advancements in the field.

Summary updated

BibTeX
@inproceedings{peng2023omnievent,
  author = {Hao Peng and Xiaozhi Wang and Feng Yao and Zimu Wang and Chuzhao Zhu and Kaisheng Zeng and Lei Hou and Juanzi Li},
  title = {{OmniEvent: A Comprehensive, Fair, and Easy-to-Use Toolkit for Event Understanding}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-demo.46},
}
Figure