Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling

Yu Zhao, Hao Fei, Yixin Cao, Bobo Li, Meishan Zhang, Jianguo Wei, Min Zhang, Tat-Seng Chua

Abstract

Video Semantic Role Labeling (VidSRL) is a fundamental task in computer vision that involves decomposing video sequences into structured events by identifying predicates, participants, and their semantic relationships. Current approaches often struggle with fine-grained spatial perception and long-range temporal consistency. This paper introduces HostSG, a hierarchical framework that constructs Holistic Spatio-Temporal Scene Graphs to bridge the gap between low-level visual features and high-level event semantics. By utilizing a novel temporalized scene graph structure and an iterative refinement process based on the Graph Information Bottleneck principle, HostSG achieves state-of-the-art performance on the VidSRL benchmark.

1. Introduction

Understanding "who did what to whom" in a video requires more than simple action recognition; it necessitates a structured understanding of the participants (arguments) and their roles relative to a predicate. This task, known as Video Semantic Role Labeling (VidSRL), is critical for applications ranging from automated video captioning to intelligent surveillance. However, existing models frequently fail to capture the specific objects involved in an event or lose track of these objects as they move across temporal boundaries. HostSG addresses these challenges by organizing video data into a hierarchical graph that preserves both spatial detail and temporal continuity.

2. Problem Statement

The authors identify two primary limitations in the current VidSRL landscape:

  1. Coarse Spatial Perception: Many models rely on global, frame-level features. This lack of granularity makes it difficult to distinguish between multiple objects in a scene (e.g., identifying which specific person is performing an action when several are present).
  2. Temporal Fragmentation: Videos are often processed as independent clips. This approach fails to account for "causal continuity," where an object's state in an early segment of the video directly influences its role in a later segment.

3. Methodology

HostSG introduces a five-tier pipeline designed to transform raw video input into a structured semantic representation.

3.1 HostSG Construction and Temporalization

The process begins by extracting keyframes and parsing them into Dynamic Scene Graphs (DSGs). To optimize computational efficiency, the model distinguishes between "static" and "dynamic" nodes. Static background elements (e.g., furniture, buildings) are merged across time, while dynamic objects (e.g., people, vehicles) maintain motion edges. Crucially, the model introduces "coreference edges" to link identical objects across different video clips, ensuring a unified narrative thread.

3.2 The ICE Graph and Spatio-Temporal Propagation

The ICE (hierarchical scene-event) graph maps low-level scene nodes to high-level event nodes (e.g., mapping a "man" node to an "Agent" role). Information is then propagated through this structure using a dual-mechanism approach:

  • Multipath-GAT (Graph Attention Network): Processes spatial relationships and local features within individual clips.
  • GGNN (Gated Graph Sequence Neural Network): Facilitates the flow of information across the temporal dimension of the entire video sequence.

3.3 Iterative Structure Refinement via GIB

Automated scene graph parsers often introduce noise. To mitigate this, HostSG employs the Graph Information Bottleneck (GIB) principle. This allows the model to adaptively adjust edge weights, pruning irrelevant visual clutter and strengthening connections that are most informative for the final prediction task. This refinement ensures that the graph structure is optimized for semantic accuracy rather than just visual reconstruction.

3.4 Joint Decoding

To prevent the error propagation common in pipeline-based methods, HostSG utilizes a joint decoding strategy. It simultaneously predicts the verb, generates descriptive text for arguments, and classifies event relations, allowing the model to leverage dependencies between these tasks.

4. Experimental Evaluation

4.1 Dataset

The model was evaluated on the VidSRL dataset, which contains 28,000 movie-sourced videos. Each 10-second video is segmented into five 2-second clips, annotated with over 2,000 verb categories and complex semantic roles. The underlying scene graph nodes were initialized using the MOTIFS-TDE parser.

4.2 Results

HostSG demonstrated superior performance across multiple metrics compared to previous state-of-the-art (SoTA) models:

  • Verb Classification: Achieved 56.15% Acc@1, a significant improvement over the previous best of ~53%.
  • Semantic Role Labeling: On the CIDEr metric, HostSG reached 55.09, outperforming the previous SoTA, VideoWhisperer, by 2.79 points.
  • Coreference Resolution: The model saw its most substantial gain in the Lea metric, increasing by +5.22 points. This validates the effectiveness of the cross-clip coreference edges in maintaining object identity over time.

5. Discussion and Conclusion

HostSG represents a significant step forward in structured video understanding. By merging static nodes and utilizing coreference edges, the model reduces redundancy while enhancing narrative coherence. The application of the GIB principle further ensures that the model remains robust against the noise inherent in automated scene parsing.

Despite its success, the model's reliance on pre-trained parsers remains a bottleneck; if an object is missed during the initial parsing stage, the ICE graph cannot currently recover it. Future research will focus on end-to-end scene graph generation tailored specifically for VidSRL and extending the framework to handle longer-duration video sequences.

Summary updated

BibTeX
@inproceedings{zhao2023constructing,
  author = {Yu Zhao and Hao Fei and Yixin Cao and Bobo Li and Meishan Zhang and Jianguo Wei and Min Zhang and Tat-Seng Chua},
  title = {{Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labeling}},
  booktitle = {ACM Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=oKiSk-pMbnU},
}
Figure