Multi-Layered Scene Graphs for Spatial Perception

47 concepts39 papers1 proposal

Traditional robotic mapping relies heavily on flat, dense representations like point clouds, occupancy grids, or raw geometric primitives. While these maps are excellent for local obstacle avoidance, they scale poorly and lack semantic meaning. As Hardik Shah et al. note in ForesightNav: Learning Scene Imagination for Efficient Exploration, flat grid-based memories lack the organizational efficiency required to manage large-scale environmental data. A robot looking at a raw point cloud of a chair sees only a cluster of 3D coordinates, missing the functional context of the object and its relationship to the room.

Early work by Akira Utsumi et al. in Estimating Surface and Spatial Structure from Wire-Frame Model Using Geometrical & Heuristical Relation recognized that transforming fragmented geometric lines into a structured hierarchy of environments, objects, and surfaces is essential for high-level spatial reasoning. To bridge the gap between low-level metric data and high-level conceptual understanding, modern spatial perception organizes environments into multi-layered hierarchical scene graphs.

The Architecture of a Hierarchical Scene Graph

A hierarchical scene graph decomposes a continuous 3D environment into discrete, nested layers of abstraction. In a foundational formulation, Iro Armeni et al. introduced the 3D Scene Graph, a unified structure that grounds semantic entities—such as objects, rooms, and entire buildings—directly within a 3D spatial framework.

[Building / Floor Level]
          │
   [Functional Rooms / Zones]
          │
   [Object Instances]
          │
[Low-Level Geometry / Keyframes]

Typically, these architectures consist of four or more distinct layers:

  1. Metric/Keyframe Layer: The bedrock of the graph, containing raw sensor observations, point clouds, or camera keyframes (U. V. B. L. Udugama et al., Mono-hydra: Real-time 3D scene graph construction from monocular camera input with IMU).
  2. Geometric/Structural Layer: Structural boundaries such as walls, floors, and ceilings (Hriday Bavle et al., S-Graphs 2.0 - A Hierarchical-Semantic Optimization and Loop Closure for SLAM).
  3. Object/Instance Layer: Segmented 3D objects and their constituent parts, capturing local geometry and attributes (Ruiyuan Lyu et al., MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations).
  4. Functional/Topological Layer: Rooms, zones, and floor levels that define spatial connectivity and human-centric boundaries (Sixian Zhang et al., Hierarchical Object-to-Zone Graph for Object Navigation).

By structuring space this way, the representation mirrors how humans conceptualize environments. This allows models to learn semantic relationships, such as the association between a bed and its constituent components like pillows and duvets, as highlighted by Akshay Gadi Patil et al. in Advances in Data-Driven Analysis and Synthesis of 3D Indoor Scenes.

Why It Works: Multi-Level Optimization

The primary computational advantage of a hierarchical scene representation is its ability to perform joint reasoning and multi-level optimization. In traditional Simultaneous Localization and Mapping (SLAM), sensor drift accumulates over time, leading to distorted maps. Hierarchical graphs solve this by using high-level semantic relationships to guide the refinement of low-level geometric data.

In S-Graphs 2.0, Hriday Bavle et al. demonstrate that tracking structural elements like walls and rooms allows the system to perform floor-aware loop closures. If the system knows that two keyframes belong to the same room, it can enforce a high-level topological constraint that optimizes the entire underlying trajectory, resulting in faster and more accurate state estimation.

This abstraction also makes global localization highly efficient. Muhammad Shaheer, in Graph-based Global Robot Localization Informing Situational Graphs with Architectural Graphs, demonstrates that matching an online-constructed situational graph against an architectural Building Information Model (BIM) graph makes global localization robust to environmental changes. By shifting the problem from raw point-cloud matching to high-level graph matching, the system bypasses the need for expensive geometric computations.

For an embodied agent executing complex tasks, a hierarchical map transforms an intractable global search problem into an efficient, localized one. Instead of scanning every cubic centimeter of a house to find an item, the agent reasons top-down.

Sixian Zhang et al. introduce this intuition via the Hierarchical Object-to-Zone Graph for Object Navigation, where clusters of related objects are grouped into intermediate "zones." An agent can navigate efficiently to a broad zone (e.g., the dining area) before fine-tuning its actions to locate a specific target (e.g., a fork). This progressive narrowing of the search space—moving from floors to rooms to fine-grained visual targets—is also utilized by Ning Lan et al. in Visual Environment-Interactive Planning for Embodied Complex-Question Answering to answer multi-step questions without suffering from planning hallucinations.

[Global Scene Overview] ──> [Functional Zone] ──> [Object Instance] ──> [Component Part]

Furthermore, these hierarchies capture functional and behavioral priors. As Andrey Kurenkov et al. show in Semantic and Geometric Modeling with Neural Message Passing in 3D Scene Graphs for Hierarchical Mechanical Search, neural message passing across a scene graph allows a robot to infer where hidden objects reside based on semantic context (e.g., tea is likely inside a cabinet in the kitchen), minimizing the mechanical actions required during a search.

Grounding Language and Human-Centric Interaction

Finally, hierarchical scene graphs serve as the critical translation layer between metric sensor data and human language. Humans do not give instructions in coordinates; they use functional regions and object relationships.

Early work by Hendrik Zender et al. in An Integrated Robotic System for Spatial Understanding and Situated Interaction in Indoor Environments proved that integrating metric, topological, and conceptual layers allows a robot to infer abstract room types (like a "living room") based on the objects detected within them.

In the era of Large Language Models (LLMs), this structural grounding has become even more vital. Dicong Qiu et al. in Open-vocabulary Mobile Manipulation in Unseen Dynamic Environments with 3D Semantic Maps demonstrate that combining 3D semantic maps with hierarchical region labels allows LLMs to execute open-vocabulary instructions in entirely novel environments. By organizing the world into logical units based on semantic connectivity rather than simple physical proximity, as explored by Zihan Wang in NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM, agents avoid misidentifying room boundaries and can seamlessly map high-level user demands to specific physical locations.

Key Foundational Work

Estimating Surface and Spatial Structure from Wire-Frame Model Using Geometrical & Heuristical Relation (1992) — contributed 2 concept elements to this direction.
OntoNav: A Semantic Indoor Navigation System (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_scene_representation_c06cbf0a,
  title = {hierarchical scene representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-scene-representation/rdr_c06cbf0afbf64e6e9b9512f72ae41e7b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_scene_representation_c06cbf0a_v1,
  title = {hierarchical scene representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-scene-representation/rdr_c06cbf0afbf64e6e9b9512f72ae41e7b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake