Mono-hydra: Real-time 3D scene graph construction from monocular camera input with IMU

U. V. B. L. Udugama, G. Vosselman, F. Nex

Abstract

The ability of robots to autonomously navigate through 3D environments depends on their comprehension of spatial concepts, ranging from low-level geometry to high-level semantics, such as objects, places, and buildings. To enable such comprehension, 3D scene graphs have emerged as a robust tool for representing the environment as a layered graph of concepts and their relationships. However, building these representations using monocular vision systems in real-time remains a difficult task that has not been explored in depth. This paper puts forth a real-time spatial perception system Mono-Hydra, combining a monocular camera and an IMU sensor setup, focusing on indoor scenarios. The system employs a suite of deep learning algorithms to derive depth and semantics and uses a robocentric visual-inertial odometry (VIO) algorithm based on square-root information. This system achieves sub-20 cm error in real-time processing at 15 fps, enabling real-time 3D scene graph construction using a laptop GPU (NVIDIA 3080). We make Mono-Hydra publicly available at: https://github.com/UAV-Centre-ITC/Mono_Hydra.

Introduction

For a robot to navigate complex indoor environments autonomously, it requires more than a simple point cloud; it needs a structured "mental model." 3D Scene Graphs provide this by organizing the environment into hierarchical layers, ranging from raw metric meshes to objects, rooms, and entire buildings.

Hierarchical 3D scene graph representation showing layers from buildings (L1) down to objects (L5).

Existing frameworks typically rely on heavy or power-hungry depth sensors like LiDAR or RGB-D cameras. These are often impractical for small, agile platforms such as Micro Air Vehicles (MAVs). Monocular cameras offer a lightweight alternative but lack direct depth information. Mono-Hydra bridges this gap by constructing high-level semantic maps in real-time from a single 2D video stream and an IMU.

Methods

Mono-Hydra integrates several advanced modules to process monocular input into a structured 3D representation. The system architecture is designed for real-time performance by offloading computationally intensive tasks to the GPU while maintaining core logic on the CPU.

System architecture of Mono-Hydra, highlighting the parallelization between GPU-based deep learning and CPU-based graph construction.

Monocular Depth and Semantics

Since monocular cameras do not provide depth, Mono-Hydra utilizes self-supervised deep learning networks. The researchers evaluated DistDepth, which uses structure distillation for metric accuracy, and LiteMono, a lightweight architecture optimized for speed. Simultaneously, an HRNetV2 model performs semantic segmentation on RGB frames, identifying 150 classes (mapped to 20 essential categories like walls and furniture).

Robocentric Visual-Inertial Odometry (RVIO2)

To track the robot's pose, the system employs a Visual-Inertial Odometry algorithm based on square-root information. By operating in a local "robocentric" frame, it avoids the observability mismatches common in world-centric systems, ensuring consistent tracking even during agile maneuvers.

The Hydra Framework

The predicted depth, semantics, and odometry are fed into the Hydra framework. Hydra clusters this data into a hierarchical graph in real-time, generating a metric-semantic mesh that serves as the foundation for higher-level spatial reasoning.

Key Insights

  • Parallelized Processing: A critical design choice was splitting the workload: the GPU handles deep learning inference (depth and semantics), while the CPU manages VIO and graph construction.
  • Semantic Bottlenecks: Semantic segmentation was identified as the primary bottleneck, limiting the system to approximately 15 FPS. Synchronizing the faster depth estimation (~30 FPS) with semantics is vital for a coherent 3D mesh.
  • Stability through Robocentric VIO: Local frame formulation provides numerical stability and robustness against the rapid rotations typical of drone flight.

Results and Evaluation

The system was evaluated using the uHumans2 synthetic dataset and real-world data from the ITC Building. In real-world tests, Mono-Hydra achieved a Mean Error (ME) of 0.19m using the DistDepth network. The system successfully maintained a processing rate of 15 FPS on an NVIDIA RTX 3080 GPU, demonstrating its viability for real-time applications. The resulting 3D meshes were geometrically sound and correctly labeled, enabling automated detection of rooms and objects.

Conclusion

Mono-Hydra demonstrates that monocular vision, when combined with IMU data and deep learning, is sufficient for real-time 3D scene graph construction. Future work will focus on improving semantic quality, leveraging temporal consistency to reduce flickering, and optimizing the system for embedded deployment on low-power hardware. Additionally, enhancing loop closure detection using the hierarchical descriptors of the scene graph remains a key objective.

Summary updated

Abstract

The ability of robots to autonomously navigate through 3D environments depends on their comprehension of spatial concepts, ranging from low-level geometry to high-level semantics, such as objects, places, and buildings. To enable such comprehension, 3D scene graphs have emerged as a robust tool for representing the environment as a layered graph of concepts and their relationships. However, building these representations using monocular vision systems in real-time remains a difficult task that has not been explored in depth. This paper puts forth a real-time spatial perception system Mono-Hydra, combining a monocular camera and an IMU sensor setup, focusing on indoor scenarios. However, the proposed approach is adaptable to outdoor applications, offering flexibility in its potential uses. The system employs a suite of deep learning algorithms to derive depth and semantics. It uses a robocentric visual-inertial odometry (VIO) algorithm based on square-root information, thereby ensuring consistent visual odometry with an IMU and a monocular camera. This system achieves sub-20 cm error in real-time processing at 15 fps, enabling real-time 3D scene graph construction using a laptop GPU (NVIDIA 3080). This enhances decision-making efficiency and effectiveness in simple camera setups, augmenting robotic system agility. We make Mono-Hydra publicly available at: https://github.com/UAV-Centre-ITC/Mono_Hydra.
BibTeX
@inproceedings{udugama2023mono,
  author = {U. V. B. L. Udugama and G. Vosselman and F. Nex},
  title = {{Mono-hydra: Real-time 3D scene graph construction from monocular camera input with IMU}},
  booktitle = {Geospatial Week , Vol. 10-1},
  year = {2023},
  url = {https://openreview.net/forum?id=D3nAobnveg},
}
Figure