(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering

Anoop Cherian, Chiori Hori, Tim K. Marks, Jonathan Le Roux

Abstract

Video Question Answering (VideoQA) requires sophisticated reasoning over spatio-temporal events. Traditional 2D scene graph approaches suffer from high redundancy and a lack of 3D spatial awareness. We present a (2.5+1)D spatio-temporal scene graph representation that lifts 2D frames into a pseudo-3D space. By segregating static background elements from dynamic actors and employing a hierarchical Transformer with spatio-temporal kernels, our method achieves state-of-the-art performance on NExT-QA and AVSD-QA benchmarks while significantly reducing computational overhead and training time.

Introduction

Most current VideoQA systems rely on building scene graphs—networks of objects and their relationships—for every individual frame of a video. This approach is fundamentally limited by two factors: redundancy and a lack of 3D context. Processing a static object, such as a table, across dozens of frames is computationally wasteful. Furthermore, 2D graphs struggle with object permanence and occlusions, failing to recognize that an object moving behind an obstacle remains the same entity.

Researchers from Mitsubishi Electric Research Labs (MERL) propose that videos should be treated as 2D projections of a 3D world. To reason effectively about complex "why" or "how" questions, a model must distinguish between the "stage" (static background) and the "actors" (dynamic objects) within a physically grounded 3D structure.

Methodology

The proposed pipeline transforms video frames into a more efficient and physically grounded representation through several stages:

  1. (2.5+1)D Construction: Using a depth estimation model (MiDAS), 2D object coordinates are "lifted" into a 3D world coordinate system by assigning a depth value to every detected object.
  2. Static vs. Dynamic Segregation: Objects are categorized by class. Static objects (e.g., furniture) are merged across frames if they occupy the same 3D space, drastically reducing node count. Dynamic objects (e.g., people) are maintained as sequences to preserve trajectories.
  3. Motion Enrichment: Dynamic nodes are enriched with I3D motion features to capture specific actions.
  4. Hierarchical (2.5+1)D Transformer: A novel Transformer architecture uses spatio-temporal kernels to calculate similarity based on 3D proximity and time. Multiple kernel scales allow the model to reason about both immediate interactions and long-term event evolutions.
Overview of the (2.5+1)D spatio-temporal scene graph pipeline, from input frames to answer selection.

Experimental Results

The model was evaluated on two challenging benchmarks: NExT-QA, which focuses on high-level causal and temporal explanations, and AVSD-QA, which requires reasoning over video and conversational history.

Performance and Efficiency

The (2.5+1)D representation successfully pruned 24% to 54% of graph nodes, leading to significant efficiency gains. The model trained up to four times faster than versions using unpruned 2D graphs. In terms of accuracy, the model achieved 53.4% on NExT-QA, a 4% improvement over previous state-of-the-art methods like HGA. The most notable gains were in "Why" (+5.4%) and "Location" (+3.5%) categories, highlighting the strength of spatio-temporal grounding.

Qualitative comparison of the proposed model against baselines, demonstrating superior reasoning on complex causal and descriptive questions.

Discussion and Conclusion

A key insight of this work is that 3D grounding allows for effective node pruning and helps disentangle camera motion from object motion. By registering objects in a shared 3D space, the model maintains object permanence more effectively than 2D-only approaches.

While the current model relies on pseudo-3D depth estimation and heuristic object segregation, it sets a new standard for efficient VideoQA. Future work may involve learning dynamic behavior identification on the fly and integrating audio-visual cues to further enhance event localization in 3D environments.

Summary updated

Abstract

Spatio-temporal scene-graph approaches to video-based reasoning tasks, such as video question-answering (QA), typically construct such graphs for every video frame. These approaches often ignore the fact that videos are essentially sequences of 2D "views" of events happening in a 3D space, and that the semantics of the 3D scene can thus be carried over from frame to frame. Leveraging this insight, we propose a (2.5+1)D scene graph representation to better capture the spatio-temporal information flows inside the videos. Specifically, we first create a 2.5D (pseudo-3D) scene graph by transforming every 2D frame to have an inferred 3D structure using an off-the-shelf 2D-to-3D transformation module, following which we register the video frames into a shared (2.5+1)D spatio-temporal space and ground each 2D scene graph within it. Such a (2.5+1)D graph is then segregated into a static sub-graph and a dynamic sub-graph, corresponding to whether the objects within them usually move in the world. The nodes in the dynamic graph are enriched with motion features capturing their interactions with other graph nodes. Next, for the video QA task, we present a novel transformer-based reasoning pipeline that embeds the (2.5+1)D graph into a spatio-temporal hierarchical latent space, where the sub-graphs and their interactions are captured at varied granularity. To demonstrate the effectiveness of our approach, we present experiments on the NExT-QA and AVSD-QA datasets. Our results show that our proposed (2.5+1)D representation leads to faster training and inference, while our hierarchical model showcases superior performance on the video QA task versus the state of the art.
BibTeX
@inproceedings{cherian2022d,
  author = {Anoop Cherian and Chiori Hori and Tim K. Marks and Jonathan Le Roux},
  title = {{(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=M3c5T2DconF},
}
Figure