Spatio-Temporal Scene Graphs for Dynamic Scene Understanding

This direction is split into other directions: temporal relationship modeling. Its last version stays readable and citable here.

Version 1 of 1295 papers309 conceptsfrozen 2026-10-04

Traditional computer vision treats the world as a series of static snapshots. While standard scene graphs—structured representations where nodes are objects and edges are their relationships—have excelled at describing single images, they struggle with the fluid reality of the physical world. Research is shifting toward Spatio-Temporal Scene Graphs, a framework that treats environment understanding not as a single state, but as an evolving narrative of interactions.

From Snapshots to Sequences

The fundamental intuition behind this shift is that actions are not monolithic labels; they are compositions of changing relationships. In Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs, Jingwei Ji and Ranjay Krishna argue that an action like "opening a drawer" is best understood as a sequence of graphs where the relationship between a hand and a handle transitions from approaching to touching to pulling.

By decomposing video into these structured transitions, models can achieve better few-shot learning and more granular action recognition. However, as noted by Anna Rohrbach and Trevor Darrell in their work on Language-Conditioned Graph Networks, the challenge lies in maintaining "context-aware" representations that can track these evolving relationships across frames without losing the global thread of the scene.

The Challenge of Temporal Consistency

In a dynamic environment, the hardest task for a graph-based model is identity maintenance. If a robot moves an object or a person walks behind a pillar, the scene graph must update its edges without fragmenting the nodes. Yuhong Deng and Huaping Liu highlight this in their work on Embodied Exploration, where a robot must "clear clutter" to answer questions. If the graph doesn't realize that the "keyboard" it saw three seconds ago is the same one now partially obscured by a bin, its reasoning fails.

To solve this, researchers like Yue Qiu have proposed models like SGTracker, introduced in the VirtualHome Action Genome project. SGTracker uses a transformer-based architecture to explicitly leverage temporal context, allowing the model to distinguish between visually similar but temporally distinct actions, such as "opening" versus "closing" a cabinet, by looking at the trajectory of the relationship over time.

Hierarchies and Open Vocabularies

As scene graphs move into 3D and robotics, they are becoming increasingly hierarchical and semantically flexible. Johanna Wald and Federico Tombari demonstrated in Learning 3D Semantic Scene Graphs that learning joint instance embeddings directly from 3D point clouds allows for complex spatial reasoning (e.g., "bigger than," "standing on").

Building on this, Qiao Gu and colleagues introduced ConceptGraphs, which fuses 2D foundation models (like CLIP and SAM) into 3D scene graphs. This "open-vocabulary" approach means a robot doesn't just see a "node"; it can reason about that node using natural language, such as "find me something to wear to a space party."

To manage the complexity of these environments, Wenting Xu and Viorela Ila developed TB-HSU, which organizes the graph into layers:

  1. Objects: Individual entities.
  2. Regions: Functional areas (e.g., a workspace).
  3. Rooms: High-level semantic containers.

Reasoning with Hyper-Edges

The most recent evolution in this field involves moving beyond simple binary edges (Subject-Predicate-Object) to more complex structures. Bo Wu introduced Situation Hyper-Graphs (SHG-VQA) in Learning Situation Hyper-Graphs for Video Question Answering. In this view, actions act as "hyper-edges" that bridge multiple objects and relationships across time. This allows the model to capture not just what is happening, but the causal links between sequential events.

Despite these advances, a "decomposition bottleneck" remains. As shown in TSG Bench by Dongil Yang and colleagues, even powerful Large Language Models struggle to break down multi-step actions into the discrete, structured snapshots required for robotics. The future of the field lies in bridging this gap—creating models that can autonomously generate and update these complex, symbolic maps of a world in motion.

Papers in this version

Showing 60 of 295 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatio_temporal_scene_graphs_for_dynamic_20a0a9e8,
  title = {Spatio-Temporal Scene Graphs for Dynamic Scene Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatio-temporal-scene-graphs-for-dynamic-scene-understanding/rdr_20a0a9e8005c42e3856c3e9327447729},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatio_temporal_scene_graphs_for_dynamic_20a0a9e8_v1,
  title = {Spatio-Temporal Scene Graphs for Dynamic Scene Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatio-temporal-scene-graphs-for-dynamic-scene-understanding/rdr_20a0a9e8005c42e3856c3e9327447729/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}