Abstract
We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling. We represent videos as a set of related events, wherein each event consists of a verb and multiple entities that fulfill various roles relevant to that event. To study the challenging task of semantic role labeling in videos or VidSRL, we introduce the VidSitu benchmark, a large-scale video understanding data source with $29K$ $10$-second movie clips richly annotated with a verb and semantic-roles every $2$ seconds. Entities are co-referenced across events within a movie clip and events are connected to each other via event-event relations. Clips in VidSitu are drawn from a large collection of movies (${\sim}3K$) and have been chosen to be both complex (${\sim}4.2$ unique verbs within a video) as well as diverse (${\sim}200$ verbs have more than $100$ annotations each). We provide a comprehensive analysis of the dataset in comparison to other publicly available video understanding benchmarks, several illustrative baselines and evaluate a range of standard video recognition models. Our code and dataset is available at vidsitu.org.
Introduction: Beyond Action Classification
Current video understanding research typically bifurcates into two paradigms: action classification and video captioning. Action classification, while providing clear labels like "running" or "punching," often fails to capture the nuanced context of "who" is performing the action, "where" it is happening, and "how" it relates to other events. Conversely, video captioning offers natural language descriptions but lacks a formal structure, making objective evaluation difficult.
To address these limitations, we introduce Visual Semantic Role Labeling (VidSRL). This framework represents videos as a series of structured, interconnected events. Each event is defined by a salient verb and the entities that fulfill specific semantic roles (e.g., Agent, Patient, Instrument). By structuring video understanding in this way, we can capture the complex narrative flow of a scene.
The VidSitu Dataset
To facilitate research in VidSRL, we present VidSitu, a large-scale benchmark consisting of 29,000 10-second movie clips sourced from approximately 3,000 different films. The dataset is uniquely designed for complexity and diversity:
- Event Density: Each 10-second clip is annotated with five events, one every 2 seconds.
- Semantic Richness: Annotations utilize a curated set of ~2.2K verb senses from PropBank. On average, each video contains 4.2 unique verbs and 6.5 unique entities.
- Entity Co-reference: Entities are tracked across events within a clip, requiring models to maintain consistent identification (e.g., recognizing that the "woman with shield" in Event 1 is the same entity in Event 5).
- Event Relations: Events are connected via semantic relations such as causality, enablement, or reaction, providing a higher-level understanding of the video's narrative.
Compared to existing datasets like MSR-VTT or ActivityNet-Captions, VidSitu features significantly higher verb density and more explicit annotations for entity persistence over time.
The VidSRL Framework: Modeling Events and Relations
We decompose the VidSRL task into three primary sub-challenges, providing a baseline pipeline to address each:
- Verb Prediction: The model identifies the most salient verb for each 2-second segment. We evaluate standard video backbones like I3D and SlowFast for this task.
- Argument Prediction & Co-referencing: Given the video and a predicted verb, a Transformer-based encoder-decoder generates text for semantic roles. To handle co-reference, the model must use consistent string identifiers for the same entity across different time steps.
- Event Relation Prediction: This stage classifies the relationship between events. We employ a RoBERTa language model to encode the structured text (verbs and roles) alongside visual features to determine how events enable or react to one another.
Experimental Results and Insights
Our evaluation reveals a significant performance gap between state-of-the-art models and human performance. In semantic role prediction, humans achieve a CIDEr score of ~84, whereas the best-performing models reach only ~48.
Key findings from our experiments include:
- The Necessity of Visual Context: Models attempting to predict event relations using only text (verbs and roles) without visual input failed to outperform a simple "most frequent class" baseline. This underscores that visual cues are indispensable for understanding causality and temporal logic in video.
- Architecture Strengths: While SlowFast models excelled at verb prediction, I3D models showed a slight advantage in predicting semantic roles, suggesting that different architectures may be better suited for "what" (actions) versus "who/how" (roles).
- Impact of Pre-training: Pre-training on the large-scale Kinetics dataset significantly improved verb prediction recall (by 10-16 points), demonstrating the importance of broad action recognition capabilities for structured tasks.
Conclusion and Future Work
VidSitu and the VidSRL framework offer a more granular and structured approach to video understanding than traditional methods. However, several challenges remain. The current 2-second fixed interval is a heuristic that may not capture actions of varying durations. Furthermore, while roles are currently represented as text, future work could involve "spatial grounding"—linking these roles to specific bounding boxes or segments within the video pixels. By moving toward structured, relation-aware representations, we can develop models that truly understand the stories told in video.
Summary updated