Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks

Zachary Ravichandran, Lisa Peng, Nathan Hughes, J. Daniel Griffith, Luca Carlone

TL;DR

This paper introduces a novel reinforcement learning approach for robot navigation that leverages hierarchical 3D scene graphs and explicit memory. By representing environments as structured graphs, robots can learn more efficient and robust navigation policies, outperforming traditional methods in complex multi-object search tasks.

What Problem Does This Paper Address?

Learning effective navigation policies for robots is a significant challenge. While using mid-level perceptual abstractions like depth or semantic segmentation as input to policies has improved sample efficiency compared to raw sensor data, these policies still struggle to grasp higher-level, three-dimensional scene properties. Robots need to understand not just what's around them, but also the spatial relationships, topology, and semantics of their environment to navigate effectively, especially in complex tasks like searching for objects. Existing methods often require pre-built maps or struggle with long-term memory, leading to inefficient exploration and revisits.

Methods

The core of this work is a reinforcement learning framework that uses 3D Dynamic Scene Graphs (DSGs) as the robot's representation of the environment. A DSG is a hierarchical graph that captures geometry, topology, and semantics at different levels of abstraction (e.g., mesh, objects, places, rooms, buildings).

Key methodological contributions include:

  • Graph Neural Network (GNN) Policy: A GNN is employed to process the DSG and learn a navigation policy. This allows the robot to reason about the relationships between different entities in the scene.
  • Agent-Centric Embedding: The global DSG is transformed into an agent-centric feature space. This enables the learning of platform-agnostic policies, meaning the learned policy can generalize to different robot platforms or control spaces (e.g., "move forward," "turn left").
  • Explicit Memory Integration: The robot's trajectory is explicitly incorporated into the graph representation. This allows the policy to learn from its past movements and avoid revisiting already explored areas.
  • Action Layer: A novel "Action layer" is added to the graph observation. This layer represents the robot's immediate surroundings and action space, acting as a bridge between the high-level scene graph and low-level control commands. This layer is crucial for connecting the learned policy to executable actions.
  • Occupancy Checking: The connection between the Action layer and the DSG incorporates occlusion checking to ensure that action nodes are placed in traversable free-space, preventing the policy from learning unsafe behaviors.

Datasets

The experiments were conducted in a photo-realistic Unity-based simulator of indoor office buildings. The simulator provided ground-truth data including RGB imagery, 2D semantic segmentation, depth, and an ESDF (Euclidean Signed Distance Function) slice. The environments were split into training and testing sets, with a total area of 8,730 m². No new datasets were introduced; rather, the simulator provided a rich environment for evaluating the proposed method.

Key Insights and Ideas

  • Hierarchy is Key: Moving beyond mid-level abstractions to high-level, hierarchical representations like 3D scene graphs significantly improves navigation performance. The structure of the DSG allows the robot to reason about where targets are likely to be found, rather than just reacting to immediate sensory input.
  • Explicit Memory Pays Off: Incorporating the robot's trajectory history into the representation is crucial for efficient exploration. Without it, robots tend to revisit areas, wasting valuable time and reducing the overall number of targets found.
  • The Action Layer is a Critical Link: The proposed "Action layer" is vital for translating the abstract graph representation into concrete navigation actions. Without it, the robot struggles to map the scene graph to its low-level control commands.
  • Semantic Noise is Challenging but Manageable: While semantic noise in the DSG degrades performance, the proposed method still outperforms baselines that don't leverage hierarchical information. This suggests the policy can still extract useful signals even with imperfect semantic data, but it highlights the importance of accurate scene graph construction.
  • Position Noise Leads to Conservative Behavior: When node positions in the DSG are perturbed, the robot becomes overly cautious, performing tight maneuvers and exploring less. This indicates the policy relies heavily on accurate spatial information for efficient exploration.

Key Results

  • Superior Performance in Multi-Object Search: The proposed method significantly outperformed baselines (RGB, RGB-D + Semantics, ESDF) in terms of targets found (44.2% vs. ~30-40%) and area explored (59.1 m² vs. ~38-53 m²).
  • Reduced Collisions (with caveats): While the proposed method had more collisions than RGB-D + Semantics and RGB baselines, it had significantly fewer collisions than the ESDF baseline. The paper notes that the sparse nature of the DSG representation might contribute to this, as it infers free-space rather than directly sensing it.
  • Ablation Studies Confirm Contributions:
    • Removing hierarchical information (using only the Places layer) led to a drop in targets found.
  • Removing explicit memory (trajectory history) resulted in the robot revisiting areas more often and exploring less.
  • Removing the Action layer drastically reduced performance, with the robot finding only 8.7% of targets.
  • Disabling occlusion checking in the Action layer also significantly increased collisions and reduced performance.

Limitations and Future Work

  • Obstacle Avoidance: While the method improves navigation, obstacle avoidance using solely graphical data remains a challenge. Future work could explore incorporating denser depth information from the DSG's mesh layer or refining the Action layer.
  • DSG Construction Noise: The paper investigates the impact of noise (position, semantic, and latency) on DSG construction. While the method shows resilience, further work could explore more robust DSG construction techniques or end-to-end learning that is less sensitive to noise.
  • Complex Environments: The current work focuses on indoor office environments. Extending to more complex, dynamic, or outdoor environments would be a valuable next step.
  • More Sophisticated GNNs and Features: The authors suggest exploring more complex GNN architectures, vector embeddings for semantic classes, and additional node attributes to capture richer information.
  • Real-World Deployment: Integrating with real-time DSG construction methods like Hydra [79] is a key future direction for enabling experiments on physical robots.

Summary updated

Abstract

Representations are crucial for a robot to learn effective navigation policies. Recent work has shown that mid-level perceptual abstractions, such as depth estimates or 2D semantic segmentation, lead to more effective policies when provided as observations in place of raw sensor data (e.g., RGB images). However, such policies must still learn latent three-dimensional scene properties from mid-level abstractions. In contrast, high-level, hierarchical representations such as 3D scene graphs explicitly provide a scene's geometry, topology, and semantics, making them compelling representations for navigation. In this work, we present a reinforcement learning framework that leverages high-level hierarchical representations to learn navigation policies. Towards this goal, we propose a graph neural network architecture and show how to embed a 3D scene graph into an agent-centric feature space, which enables the robot to learn policies that map 3D scene graphs to a platform-agnostic control space (e.g., go straight, turn left). For each node in the scene graph, our method uses features that capture occupancy and semantic content, while explicitly retaining memory of the robot trajectory. We demonstrate the effectiveness of our method against commonly used visuomotor policies in a challenging multi-object search task. These experiments and supporting ablation studies show that our method leads to more effective object search behaviors, exhibits improved long-term memory, and successfully leverages hierarchical information to guide its navigation objectives.
BibTeX
@inproceedings{ravichandran2022hierarchical,
  author = {Zachary Ravichandran and Lisa Peng and Nathan Hughes and J. Daniel Griffith and Luca Carlone},
  title = {{Hierarchical Representations and Explicit Memory: Learning Effective Navigation Policies on 3D Scene Graphs using Graph Neural Networks}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=qhtzEMBv8I},
}
Figure