Visual Semantic Role Labeling for Video Understanding

Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Nevatia, Aniruddha Kembhavi

Abstract

We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling. We represent videos as a set of related events, wherein each event consists of a verb and multiple entities that fulfill various roles relevant to that event. To study the challenging task of semantic role labeling in videos or VidSRL, we introduce the VidSitu benchmark, a large scale video understanding data source with 27K 10-second movie clips richly annotated with a verb and semantic-roles every 2 seconds. Entities are co-referenced across events within a movie clip and events are connected to each other via event-event relations. Clips in VidSitu are drawn from a large collection of movies (~3K) and have been chosen to be both complex (~4.2 unique verbs within a video) as well as diverse (~200 verbs have more than 100 annotations each). We provide a comprehensive analysis of the dataset in comparison to other publicly available video understanding benchmarks, several illustrative baselines and evaluate a range of standard video recognition models.

Introduction

Most video understanding research focuses on narrow tasks like action classification, localization, or captioning. While useful, these approaches often fail to provide a holistic understanding of complex scenes. Action labels are frequently too "impoverished," missing the who, what, where, and why. Conversely, video captions are unstructured and notoriously difficult to evaluate.

We propose bridging this gap by treating video as a series of related situations. Our framework, Visual Semantic Role Labeling (VidSRL), requires models to produce a structured representation of a scene. This involves identifying not just the primary action (the verb) but also the semantic roles (arguments) associated with that action, tracking those entities across time, and characterizing the causal relationships between sequential events.

The VidSRL Framework

The VidSRL task is formalized into three distinct sub-problems:

  1. Verb Prediction: Identifying the most salient verb for every 2-second interval.
  2. Semantic Role Prediction & Co-referencing: Predicting the entities filling roles like Arg0 (agent) or Arg1 (patient). Crucially, if the same entity appears in multiple events, the model must "co-reference" them using a consistent description.
  3. Event Relation Prediction: Determining how events relate to one another using a taxonomy including Caused By, Enabled By, Reaction To, or Unrelated.
Overview of the VidSRL task, illustrating verbs, semantic roles, and event-event relations across a 10-second clip.

The VidSitu Dataset

To support this research, we introduce the VidSitu dataset, consisting of 29,000 10-second movie clips drawn from approximately 3,000 movies. Movie clips were chosen for their narrative diversity and complex interactions between multiple entities, avoiding the visual repetitiveness found in many "in-the-wild" web datasets.

VidSitu is significantly more complex than existing benchmarks. It features a large vocabulary of ~2,200 verb senses from PropBank and maintains a high density of unique verbs and entities per video. This richness ensures that models must rely on temporal context; for instance, a character occluded in one frame may only be identifiable by looking at the rest of the 10-second clip.

Comparison of VidSitu with other benchmarks across diversity, complexity, and richness metrics.

Key Results and Analysis

We evaluated several baselines using state-of-the-art video backbones like I3D and SlowFast. Our findings highlight several key challenges:

  • The Human-Machine Gap: There is a massive performance gap in role prediction. On the CIDEr metric, humans achieve a score of ~84, while the best models hover around ~47.
  • Co-reference Struggles: Current models struggle to maintain a consistent identity for entities across different events in the same clip, often failing to link a "man" in one interval to the same person in the next.
  • Causal Reasoning: Event relation prediction proved to be the most difficult task. Adding video features provided only marginal gains over text-only baselines, suggesting that reasoning about causality in video remains a significant open challenge.

Conclusion and Future Work

VidSitu provides a comprehensive benchmark for moving beyond simple action labels toward a structured understanding of video narratives. While current models perform reasonably well at basic verb recognition, they falter at the more complex tasks of entity tracking and causal reasoning. Future work will focus on integrating spatial grounding (e.g., bounding boxes) directly into the VidSRL pipeline and extending the framework to understand longer-form narrative arcs in full-length movies.

Summary updated

Abstract

We propose a new framework for understanding and representing related salient events in a video using visual semantic role labeling. We represent videos as a set of related events, wherein each event consists of a verb and multiple entities that fulfill various roles relevant to that event. To study the challenging task of semantic role labeling in videos or VidSRL, we introduce the VidSitu benchmark, a large scale video understanding data source with 27K 10-second movie clips richly annotated with a verb and semantic-roles every 2 seconds. Entities are co-referenced across events within a movie clip and events are connected to each other via event-event relations. Clips in VidSitu are drawn from a large collection of movies ( 3K) and have been chosen to be both complex ( 4.2 unique verbs within a video) as well as diverse ( 200 verbs have more than 100 annotations each). We provide a comprehensive analysis of the dataset in comparison to other publicly available video understanding benchmarks, several illustrative baselines and evaluate a range of standard video recognition models. Our code and dataset will be released publicly.
BibTeX
@inproceedings{sadhu2021visual,
  author = {Arka Sadhu and Tanmay Gupta and Mark Yatskar and Ram Nevatia and Aniruddha Kembhavi},
  title = {{Visual Semantic Role Labeling for Video Understanding}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=zybfu9hi3dD},
}
Figure