Abstract
We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling. We represent videos as a set of related events, wherein each event consists of a verb and multiple entities that fulfill various roles relevant to that event. To study the challenging task of semantic role labeling in videos or VidSRL, we introduce the VidSitu benchmark, a large scale video understanding data source with 27K 10-second movie clips richly annotated with a verb and semantic-roles every 2 seconds. Entities are co-referenced across events within a movie clip and events are connected to each other via event-event relations. Clips in VidSitu are drawn from a large collection of movies (~3K) and have been chosen to be both complex (~4.2 unique verbs within a video) as well as diverse (~200 verbs have more than 100 annotations each). We provide a comprehensive analysis of the dataset in comparison to other publicly available video understanding benchmarks, several illustrative baselines and evaluate a range of standard video recognition models.
Introduction
Most video understanding research focuses on narrow tasks like action classification, localization, or captioning. While useful, these approaches often fail to provide a holistic understanding of complex scenes. Action labels are frequently too "impoverished," missing the who, what, where, and why. Conversely, video captions are unstructured and notoriously difficult to evaluate.
We propose bridging this gap by treating video as a series of related situations. Our framework, Visual Semantic Role Labeling (VidSRL), requires models to produce a structured representation of a scene. This involves identifying not just the primary action (the verb) but also the semantic roles (arguments) associated with that action, tracking those entities across time, and characterizing the causal relationships between sequential events.
The VidSRL Framework
The VidSRL task is formalized into three distinct sub-problems:
- Verb Prediction: Identifying the most salient verb for every 2-second interval.
- Semantic Role Prediction & Co-referencing: Predicting the entities filling roles like Arg0 (agent) or Arg1 (patient). Crucially, if the same entity appears in multiple events, the model must "co-reference" them using a consistent description.
- Event Relation Prediction: Determining how events relate to one another using a taxonomy including Caused By, Enabled By, Reaction To, or Unrelated.
The VidSitu Dataset
To support this research, we introduce the VidSitu dataset, consisting of 29,000 10-second movie clips drawn from approximately 3,000 movies. Movie clips were chosen for their narrative diversity and complex interactions between multiple entities, avoiding the visual repetitiveness found in many "in-the-wild" web datasets.
VidSitu is significantly more complex than existing benchmarks. It features a large vocabulary of ~2,200 verb senses from PropBank and maintains a high density of unique verbs and entities per video. This richness ensures that models must rely on temporal context; for instance, a character occluded in one frame may only be identifiable by looking at the rest of the 10-second clip.
Key Results and Analysis
We evaluated several baselines using state-of-the-art video backbones like I3D and SlowFast. Our findings highlight several key challenges:
- The Human-Machine Gap: There is a massive performance gap in role prediction. On the CIDEr metric, humans achieve a score of ~84, while the best models hover around ~47.
- Co-reference Struggles: Current models struggle to maintain a consistent identity for entities across different events in the same clip, often failing to link a "man" in one interval to the same person in the next.
- Causal Reasoning: Event relation prediction proved to be the most difficult task. Adding video features provided only marginal gains over text-only baselines, suggesting that reasoning about causality in video remains a significant open challenge.
Conclusion and Future Work
VidSitu provides a comprehensive benchmark for moving beyond simple action labels toward a structured understanding of video narratives. While current models perform reasonably well at basic verb recognition, they falter at the more complex tasks of entity tracking and causal reasoning. Future work will focus on integrating spatial grounding (e.g., bounding boxes) directly into the VidSRL pipeline and extending the framework to understand longer-form narrative arcs in full-length movies.
Summary updated