Structured Scene Graphs for Spatial Reasoning and Planning

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 21,372 papers1,755 conceptsfrozen 2026-10-04

Artificial intelligence often struggles with a fundamental disconnect: raw sensory data (like pixels or point clouds) is too dense for logical reasoning, while natural language is too ambiguous for precise physical interaction. A robot might understand the command "pick up the cup to the left of the plate," but translating that into a 3D coordinate requires more than just a language model; it requires a structured understanding of spatial geometry.

To bridge this gap, researchers are increasingly using Scene Graphs—a symbolic representation where objects are nodes and their spatial or semantic relationships are edges. By abstracting the physical world into a graph, models can perform complex reasoning, imagine new environments, and align human language with 3D reality.

The Symbolic Bridge

The core intuition behind this research direction is that a graph is the natural "middle ground" between vision and language. As Justin Johnson and Agrim Gupta demonstrated in Image Generation From Scene Graphs, explicitly modeling relationships like "sheep on grass" allows a model to reason about spatial arrangements far more effectively than unstructured text.

In a scene graph, an environment is no longer a collection of independent pixels. Instead, it is a structured map of triplets (Subject-Predicate-Object). This structure provides several key advantages:

  1. Disambiguation: It resolves "perspective illusions" where 2D overlaps might hide 3D depth.
  2. Compositionality: It allows for "spatial imagination," where a model can rearrange nodes to envision a new scene before acting.
  3. Interpretability: Because the graph is symbolic, humans can inspect the "logic" the AI is using to make a decision.

From 3D Layouts to 4D Dynamics

While early work focused on 2D images, recent breakthroughs have extended this to 3D and even 4D (3D + time). Jingkang Yang and colleagues introduced PSG-4D, which tracks how these relationships evolve over time. This is crucial for tasks like surgical phase recognition, as explored by Ege Özsoy in Holistic OR domain modeling, where the movement of tools relative to anatomy defines the progress of a procedure.

For robotics, this representation is transformative. Guangyao Zhai and Yan Di developed SG-Bot, a framework that allows a robot to "imagine" a tidy version of a room by manipulating a scene graph. By moving nodes in the graph, the robot calculates the precise 3D coordinates needed to rearrange physical objects. This "coarse-to-fine" approach bridges the gap between high-level commonsense (e.g., "plates belong in the cupboard") and low-level motor control.

Grounding Language in Geometry

One of the most significant challenges in modern AI is "grounding"—ensuring that a Large Language Model (LLM) actually knows what a "fragile" object is in a physical space. Mohammed Diab proposes using scene graphs to map sensory data into foundational ontologies that represent high-level concepts like risk or fragility.

This grounding enables more sophisticated human-robot collaboration. In the SYNERGAI framework by Yixin Chen 0003 and Qing Li, 3D scene graphs act as a shared language. A human can give a preference (e.g., "don't put heavy things on the glass table"), and the robot updates its internal graph to reflect these semantic constraints.

Similarly, SpatialRGPT by An-Chieh Cheng uses 3D scene graphs to help Vision-Language Models perform "multi-hop" spatial reasoning. Instead of just identifying a chair, the model can traverse the graph to find "the chair to the right of the table that is under the lamp," a task that requires understanding the relative topology of the entire room.

Domain-Agnostic Reasoning and Simulation

Because scene graphs are symbolic, they are inherently domain-agnostic. This makes them a powerful tool for "sim-to-real" transfer. Artem Savkin and Nassir Navab demonstrated that by using 3D Scene Graphs as an intermediate step, researchers can generate realistic traffic images from simulated data without inheriting the "synthetic look" of the simulator. The graph captures the structure of the traffic (car A is behind car B), while the generative model handles the appearance.

This structural approach also allows for the discovery of "corner cases" in autonomous driving. George Drayson and Lars Kunze use learned scene graphs to manipulate relationships between pedestrians and vehicles, synthetically creating dangerous scenarios that are too rare to find in real-world data but essential for safety testing.

The Future of Scene Understanding

The shift toward structured scene graphs represents a move away from "black-box" end-to-end learning and toward neuro-symbolic architectures. By using structures like the Octree-Graph proposed by Zhigang Wang, which combines efficient 3D storage with relational reasoning, the field is moving toward models that can navigate, reason, and communicate within the complex, dynamic 3D world we inhabit.

Papers in this version

Showing 60 of 1,372 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 1309 papers
  2. Version 1 · 2026-10-04 · 1372 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_structured_scene_representation_3da839a2,
  title = {structured scene representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structured-scene-representation/rdr_3da839a2b0a641ae8307bd008851a24f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_structured_scene_graphs_for_spatial_reas_3da839a2_v1,
  title = {Structured Scene Graphs for Spatial Reasoning and Planning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/structured-scene-graphs-for-spatial-reasoning-and-planning/rdr_3da839a2b0a641ae8307bd008851a24f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}