Mono-hydra: Real-time 3D scene graph construction from monocular camera input with IMU

U. V. B. L. Udugama, George Vosselman, Francesco Nex

TL;DR

This paper introduces Mono-Hydra, a novel system that constructs real-time 3D scene graphs from a single monocular camera and an IMU. By combining deep learning for depth and semantic estimation with a robust visual-inertial odometry (VIO) algorithm, Mono-Hydra enables robots to understand their environment with unprecedented detail and speed, even with simple camera setups.

What Problem Does This Paper Address?

Robots need to understand their surroundings to navigate and interact effectively. This understanding goes beyond just geometry; it involves recognizing objects, places, and their relationships, which is often represented using 3D scene graphs. While powerful, building these detailed scene graphs in real-time, especially using only a monocular camera (which lacks inherent depth information), has been a significant challenge. Existing methods often rely on more complex sensors like RGB-D cameras or LiDAR, or they are not real-time. This paper aims to bridge this gap by enabling real-time 3D scene graph construction using a compact and cost-effective monocular camera and IMU setup.

Methods

Mono-Hydra is built upon the Hydra framework and integrates several key components:

  • Deep Learning for Depth and Semantics: The system employs deep neural networks to estimate depth from monocular images and perform semantic segmentation. This allows it to infer 3D structure and identify objects and regions within the scene.
  • Robocentric Visual-Inertial Odometry (VIO): To accurately track the robot's movement, Mono-Hydra uses a novel robocentric VIO algorithm. This approach, based on square-root information, ensures consistent odometry by treating the robot's body frame as the reference, which is crucial for maintaining spatial awareness in dynamic environments.
  • Semantic Mesh Generation: The predicted depth and semantic segmentation maps are combined to create a 3D semantic mesh. This mesh serves as the foundational geometric and semantic representation of the environment.
  • Hydra Framework Integration: The generated semantic mesh and the VIO data are fed into the Hydra framework. Hydra then uses this information to construct and optimize the hierarchical 3D scene graph, representing the environment at various levels of abstraction (e.g., buildings, rooms, objects).

The system is designed for real-time performance, with deep learning models running on a GPU and the Hydra framework utilizing the CPU.

Datasets

  • Training: The paper mentions fine-tuning depth and semantic networks on the NYUv2 (indoor) and ADE20k (general scene parsing) datasets.
  • Evaluation:
    • Synthetic Data: The uHumans2 dataset, an upgraded version of the uHumans dataset, was used for evaluating the system's performance. It provides visual-inertial data, ground truth depth, and 2D semantic segmentation.
  • Real Data: RGB sequences and IMU data were collected from the ITC building (indoor environment) for real-world testing.

Key Insights and Ideas

  • Monocular Camera Viability for Scene Graphs: The paper demonstrates that it's feasible to construct rich 3D scene graphs in real-time using only a monocular camera and IMU, overcoming a significant limitation of previous methods.
  • Synergy of Deep Learning and VIO: The combination of deep learning for perception (depth, semantics) and robust VIO for motion estimation is crucial for achieving accurate and real-time spatial understanding.
  • Robocentric VIO for Consistency: The use of a robocentric VIO approach significantly improves the consistency and robustness of motion estimation, which is vital for building accurate scene representations.
  • Hierarchical Scene Graph for Comprehensive Understanding: The Hydra framework's ability to build hierarchical scene graphs allows for a multi-level understanding of the environment, from broad structures to specific objects.

Key Results

  • Real-time Performance: Mono-Hydra achieves real-time processing at 15 fps on a laptop GPU (NVIDIA 3080).
  • Accuracy: The system achieves sub-20 cm error in real-time processing.
  • Metric Accuracy: When compared to lidar-based ground truth, the generated 3D semantic mesh shows promising metric accuracy, with Mean Errors (ME) as low as 0.03m to 0.23m using the LiteMono depth predictor on synthetic data. For real data, the DistDepth network achieved a lower Mean Error (0.19m-0.21m) compared to LiteMono (0.36m-0.39m) when compared against lidar data.
  • Semantic Segmentation Bottleneck: The semantic segmentation network was identified as the primary bottleneck, operating at approximately 15 fps, which dictated the overall system's frame rate.

Limitations and Future Work

  • Unassessed Semantic Segmentation Quality: The paper acknowledges that the quality of the semantic segmentation in the generated 3D semantic mesh was not quantitatively assessed. Future work should focus on evaluating this aspect to gain insights into the system's overall effectiveness and identify areas for optimization.
  • Hierarchical Loop Closure Optimization: Improvements are suggested in hierarchical loop closure optimization, accurate room detection, and temporal consistency of the generated 3D semantic mesh.
  • Multi-task Network Exploration: Investigating multi-task learning for depth and semantic segmentation, leveraging mutual features, is proposed.
  • Structure from Motion Integration: Exploring the integration of structure from motion techniques with monocular cameras and deep learning is another avenue for future research.
  • Deployment on Embedded Systems: While the framework shows potential for embedded systems, further optimization of deep learning networks is needed.

Summary updated

Abstract

The ability of robots to autonomously navigate through 3D environments depends on their comprehension of spatial concepts, ranging from low-level geometry to high-level semantics, such as objects, places, and buildings. To enable such comprehension, 3D scene graphs have emerged as a robust tool for representing the environment as a layered graph of concepts and their relationships. However, building these representations using monocular vision systems in real-time remains a difficult task that has not been explored in depth. This paper puts forth a real-time spatial perception system Mono-Hydra, combining a monocular camera and an IMU sensor setup, focusing on indoor scenarios. However, the proposed approach is adaptable to outdoor applications, offering flexibility in its potential uses. The system employs a suite of deep learning algorithms to derive depth and semantics. It uses a robocentric visual-inertial odometry (VIO) algorithm based on square-root information, thereby ensuring consistent visual odometry with an IMU and a monocular camera. This system achieves sub-20 cm error in real-time processing at 15 fps, enabling real-time 3D scene graph construction using a laptop GPU (NVIDIA 3080). This enhances decision-making efficiency and effectiveness in simple camera setups, augmenting robotic system agility. We make Mono-Hydra publicly available at: https://github.com/UAV-Centre-ITC/Mono_Hydra
BibTeX
@inproceedings{udugama2023mono,
  author = {U. V. B. L. Udugama and George Vosselman and Francesco Nex},
  title = {{Mono-hydra: Real-time 3D scene graph construction from monocular camera input with IMU}},
  booktitle = {Geospatial Week , Vol. 10-1},
  year = {2023},
  url = {https://openreview.net/forum?id=D3nAobnveg},
}
Figure