(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering

Anoop Cherian, Chiori Hori, Tim K. Marks, Jonathan Le Roux

TL;DR

This paper introduces a (2.5+1)D spatio-temporal scene graph representation that treats video not just as a sequence of frames, but as a series of 2D views of a 3D world. By registering objects into a shared 3D space and pruning redundant static elements, the authors achieve state-of-the-art performance in Video Question Answering (VideoQA) while speeding up training by up to 4x.

What Problem Does This Paper Address?

Most current video reasoning models process videos frame-by-frame, building a new "scene graph" (a map of objects and their relationships) for every single image. This approach has two major flaws:

  1. Redundancy: If a table sits still for 100 frames, a standard model processes it 100 times. This is computationally expensive and wasteful.
  2. Lack of 3D Context: Videos are 2D projections of a 3D world. Standard models struggle with object permanence (knowing an object is still there even if it's occluded) and camera movement because they lack a sense of depth and spatial consistency.

The authors aim to bridge this gap by creating a representation that understands the underlying 3D geometry of the scene across time.

Methods

The authors propose a multi-step pipeline to transform raw video into a compact, reasoned representation:

  1. 2D to 2.5D Transformation: They use an off-the-shelf object detector (Faster R-CNN) to find objects and a depth estimation model (MiDAS) to infer the distance of each object from the camera. This creates a "2.5D" view for every frame.
  2. 3D Registration & Pruning: Objects are categorized as Static (e.g., tables, trees) or Dynamic (e.g., people, balls). Static objects are registered into a shared 3D coordinate system across frames. If the same table appears in multiple frames, those nodes are merged into one. This significantly reduces the graph's size.
  3. Motion Enrichment: Dynamic objects, which move and interact, are enriched with motion features using an I3D action recognition model.
  4. Hierarchical (2.5+1)D Transformer: To answer questions, the model uses a novel Transformer architecture. Instead of standard attention, it uses a spatio-temporal kernel that measures how close objects are in both 3D space and time. By using multiple kernels with different "bandwidths," the model can reason at different levels of granularity—from fine-grained interactions to global scene changes.
Figure 1: A schematic illustration of our proposed (2.5+1)D video QA reasoning pipeline.

Datasets

The model was evaluated on two challenging benchmarks:

  • NExT-QA: A dataset focused on "why" and "how" questions, requiring deep causal and temporal reasoning rather than just simple object recognition.
  • AVSD-QA: Part of the Audio-Visual Scene-Aware Dialog dataset, where the model must answer questions based on video clips, often involving complex human activities.

Key Insights and Ideas

  • The Power of Pruning: By merging static nodes, the authors reduced the number of graph nodes by 24% to 54%. This isn't just an optimization; it forces the model to focus on the changes in the scene rather than getting bogged down in static background details.
  • 3D Disentanglement: Mapping objects to a shared 3D space helps the model handle camera motion. Even if the camera pans, the "table" stays at the same 3D coordinate, making it easier for the model to track interactions.
  • Hierarchical Kernels as a "Zoom" Lens: The hierarchical Transformer (shown below) allows the model to attend to objects that are very close (like a person holding a cup) while simultaneously considering broader context (like two people across a room).
Figure 2: The architecture of the proposed Hierarchical (2.5+1)D-Transformer for encoding (2.5+1)D scene graphs. The left module in red (N×) is the standard Transformer.

Key Results

  • Accuracy: The model achieved 53.4% accuracy on NExT-QA, a significant ~4% improvement over previous state-of-the-art methods like HGA.
  • Efficiency: The node pruning led to a 4x speedup in training time (from ~6 seconds per iteration down to ~1.5 seconds on an RTX 6000 GPU).
  • Reasoning Types: The model showed particularly strong gains in "Why" (+5.4%) and Location-based (+3.5%) questions, validating that the 3D spatio-temporal grounding helps with causal reasoning.
Figure 3: An example illustration of (2.5+1)D scene graphs produced by our method. The figure shows a video frame from the NExT-QA dataset, its pseduo-3D rendering, and the (2.5+1)D static and dynamic graphs computed on all frames of the video.

Limitations and Future Work

  • Dependency on Pre-trained Modules: The system's performance is capped by the accuracy of the initial 2D object detector and the depth estimation model. Errors in these early stages propagate through the pipeline.
  • Fixed Categorization: The split between "static" and "dynamic" objects is currently based on pre-defined class labels. In reality, a "static" chair can be moved. A more flexible, data-driven way to determine object mobility could be a future improvement.
  • Internet Video Challenges: The authors note that arbitrary internet videos lack camera calibration, making perfect 3D reconstruction impossible. Their "2.5D" approach is a pragmatic workaround, but true 3D reasoning remains a "holy grail" for unconstrained video.

Summary updated

Abstract

Spatio-temporal scene-graph approaches to video-based reasoning tasks, such as video question-answering (QA), typically construct such graphs for every video frame. These approaches often ignore the fact that videos are essentially sequences of 2D "views" of events happening in a 3D space, and that the semantics of the 3D scene can thus be carried over from frame to frame. Leveraging this insight, we propose a (2.5+1)D scene graph representation to better capture the spatio-temporal information flows inside the videos. Specifically, we first create a 2.5D (pseudo-3D) scene graph by transforming every 2D frame to have an inferred 3D structure using an off-the-shelf 2D-to-3D transformation module, following which we register the video frames into a shared (2.5+1)D spatio-temporal space and ground each 2D scene graph within it. Such a (2.5+1)D graph is then segregated into a static sub-graph and a dynamic sub-graph, corresponding to whether the objects within them usually move in the world. The nodes in the dynamic graph are enriched with motion features capturing their interactions with other graph nodes. Next, for the video QA task, we present a novel transformer-based reasoning pipeline that embeds the (2.5+1)D graph into a spatio-temporal hierarchical latent space, where the sub-graphs and their interactions are captured at varied granularity. To demonstrate the effectiveness of our approach, we present experiments on the NExT-QA and AVSD-QA datasets. Our results show that our proposed (2.5+1)D representation leads to faster training and inference, while our hierarchical model showcases superior performance on the video QA task versus the state of the art.
BibTeX
@inproceedings{cherian2022d,
  author = {Anoop Cherian and Chiori Hori and Tim K. Marks and Jonathan Le Roux},
  title = {{(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=M3c5T2DconF},
}
Figure