VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos

Baoyu Liang, Qile Su, Shoutai Zhu, Yuchen Liang, Chao Tong

Introduction

While computer vision has made significant strides in identifying objects and "atomic actions"—such as walking, jumping, or sitting—the field still struggles to comprehend the broader context of an event. An event is more than a simple movement; it is a structured narrative unit involving specific participants, tools, and locations, all governed by a complex web of logical relations. To bridge this gap, researchers from Beihang University have introduced VidEvent, a large-scale dataset comprising over 23,000 events designed to push AI models toward a deeper understanding of "event scripts."

The Challenge: Moving Beyond Perception to Cognition

Existing video datasets often focus on short, isolated clips or simple descriptive captions. These datasets lack the "connective tissue"—the causal, temporal, and conditional logic—that allows humans to understand a story. For an AI to move from mere perception to true cognition, it must be able to extract event scripts and reason about the long-term evolution of a scene.

VidEvent addresses this by focusing on the dynamic evolution of events, challenging models not only to identify what is happening in the moment but to induce what is likely to happen next based on narrative logic.

The VidEvent Dataset

The primary contribution of this research is the VidEvent dataset, which stands out from predecessors like VidSitu or ActivityNet in several key areas:

  • Source Material: The dataset utilizes 1,110 movie recap videos. These are considered "Goldilocks" data—natural video is often too sparse for training logical reasoning, while full-length films are too long and complex. Movie recaps condense hours of plot into minutes, making the logical progression of events highly pronounced.
  • Scale and Structure: The dataset contains 23,989 events and 17,525 relations. Annotations follow the PropBank framework, providing a standardized lexicon for verbs and their semantic roles.
  • Chain-like Evolution: Unlike datasets that only offer pairwise relations, VidEvent features continuous event chains, with an average chain length of 10.57 events, allowing for the study of long-term narrative arcs.

A Framework for Video Event Understanding (VEU)

The researchers propose a four-step pipeline to tackle the complexities of the VidEvent dataset:

  1. Video Event Localization: This involves identifying the start and end times of events. The authors introduced a Semantic Aggregation module that uses hyponym trees (hierarchical word relationships) to group short atomic actions into broader, high-level events.
  2. Video Event Extraction: Models must extract the "trigger" (the primary verb) and its "arguments" (the who, what, and where). This is achieved using an encoder-decoder architecture, utilizing models like SlowFast or TimesFormer for video and RoBERTa for text.
  3. Event Relation Classification: This task requires the model to determine how two events are linked. Relations are categorized into five types: Temporal, Causal, Conditional, Coreference, and Subevent.
  4. Script Event Induction: This is the ultimate test of reasoning. Given a sequence of events, the model must predict the most likely next event from a set of candidates using a multimodal architecture that fuses visual and textual features through cross-attention.

Key Insights and Results

The researchers established several benchmarks that highlight both the potential of the VidEvent dataset and the current limitations of AI reasoning:

  • The Hierarchy Gap: The inclusion of the semantic aggregation module significantly improved localization accuracy. This suggests that events are naturally hierarchical clusters of actions, and models must understand this nesting to find event boundaries accurately.
  • Localization Performance: Their "TriDet+Agg" model achieved a mean Average Precision (mAP) of 0.74, validating the importance of semantic grouping.
  • The Text-Vision Reasoning Gap: One of the most striking findings occurred in the Script Event Induction task. While the multimodal "TF+RoBERTa" model achieved 56% accuracy, the vision-only version of the same model plummeted to ~25%—barely better than a random guess (20%). Conversely, models using only text (RoBERTa) significantly outperformed vision-only models.

Conclusion

VidEvent reveals a major bottleneck in current AI: while visual encoders are excellent at identifying objects and actions, they are currently poor at extracting the abstract logic required for "common sense" prediction. Movie recaps provide a dense, narrative-rich environment that is ideal for training models on causality, but the performance gap between text and vision suggests that we need better visual representation learning. Future work must focus on helping visual models "read between the lines" to capture the same level of causal understanding that large language models currently derive from text.

Summary updated

BibTeX
@inproceedings{liang2025videvent,
  author = {Baoyu Liang and Qile Su and Shoutai Zhu and Yuchen Liang and Chao Tong},
  title = {{VidEvent: A Large Dataset for Understanding Dynamic Evolution of Events in Videos}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32544},
}
Figure