Monocular Object-Level SLAM Enhanced by Joint Semantic Segmentation and Depth Estimation

Ruicheng Gao, Yue QiSensors 2025 · DOI

TL;DR

This paper introduces JSDNet, a novel multi-task network for real-time joint depth estimation and semantic segmentation from monocular RGB images. This enhanced perception is then integrated into an object-level SLAM system, significantly improving localization accuracy and robustness, especially in dynamic environments, by leveraging semantic cues and a refined bundle adjustment.

What Problem Does This Paper Address?

Simultaneous Localization and Mapping (SLAM) is crucial for robots and AR/VR systems, but monocular SLAM (using only a single camera) faces significant challenges. These include scale ambiguity (difficulty in determining the true size of objects and the scene), tracking difficulties in dynamic environments, and sensitivity to lighting changes. While semantic information (understanding what objects are in a scene) can help human vision overcome these issues, traditional monocular SLAM systems struggle to effectively integrate it. Existing approaches often treat depth estimation and semantic segmentation as separate tasks or rely on RGB-D sensors, which are more expensive. This paper aims to bridge this gap by developing a unified, real-time system that leverages both depth and semantic information from a single RGB camera to achieve more accurate and robust monocular SLAM.

Methods

The core of the proposed system is JSDNet, a multi-task deep learning network designed for joint depth estimation and semantic segmentation. Key innovations in JSDNet include:

  • Depth Discretization: Instead of direct regression, depth is treated as a classification problem within a fixed, discretized range. This helps improve accuracy, especially for the critical near-to-medium range.
  • Feature Fusion Block: A novel module that allows the depth and semantic branches to share and fuse features, enabling them to mutually benefit and improve each other's performance.
  • Weight-Learned Loss Function: A homoscedastic uncertainty approach is used to balance the losses from the depth and segmentation tasks, ensuring neither dominates and both contribute effectively.
  • Semantic Consistency: A training strategy that enforces consistency between depth and semantic predictions across different views of the same scene, further improving robustness.

Building upon JSDNet, the authors design an Object-Level SLAM system. This system enhances the traditional ORB-SLAM2 pipeline by:

  • Semantic-Aware Feature Matching: Discarding features from textureless or dynamic regions (like walls, windows, or people) and points outside the estimated depth range to improve matching robustness.
  • 3D Object Representation: Representing detected objects as 3D bounding boxes, which are then integrated into the map.
  • Object-Level Bundle Adjustment (BA): Incorporating new constraints into the BA process that leverage the semantic object information (camera-object and object-point constraints) to refine camera poses and the map.
  • Scale Recovery: A dedicated process to estimate and enforce a consistent scale across the entire map, addressing the inherent scale ambiguity of monocular SLAM.

Datasets

The paper utilizes the following datasets:

  • NYU depth v2: Used for training and evaluating the JSDNet for joint depth estimation and semantic segmentation. This dataset provides RGB images, depth maps, and semantic labels.
  • TUM RGB-D: Used for evaluating the performance of the proposed object-level SLAM system, specifically for camera localization accuracy in both static and dynamic scenes.

Key Insights and Ideas

  • Synergy of Depth and Semantics: The paper strongly demonstrates that jointly learning depth and semantic segmentation is more effective than training them separately. The feature fusion and semantic consistency mechanisms are key to unlocking this synergy.
  • Semantic Information as a Robustness Enabler: By incorporating semantic understanding, the SLAM system can filter out unreliable features (e.g., from dynamic objects or areas with poor texture) and leverage object-level constraints, leading to significantly more stable and accurate localization.
  • Object-Level BA is Crucial: Adding object-specific constraints to the bundle adjustment process is shown to be highly effective in improving localization accuracy and robustness, going beyond traditional point-based BA.
  • Discretized Depth for Accuracy: Treating depth estimation as a classification problem within a defined range, rather than pure regression, yields better results, particularly for the relevant depth range for SLAM.
  • Monocular SLAM Can Compete: The work shows that with advanced semantic and depth integration, monocular SLAM can achieve performance competitive with, and in some cases superior to, RGB-D based systems, especially in dynamic scenarios.

Key Results

  • JSDNet Performance: Achieves state-of-the-art performance on the NYU depth v2 dataset for depth estimation (lowest RMSE and RMSE-Log) and competitive segmentation precision, all while maintaining real-time performance (12.1 ms per image on an Nvidia 3090 GPU).
  • SLAM Localization Accuracy: On the TUM RGB-D dataset, the proposed object-level SLAM system demonstrates significantly lower trajectory errors compared to ORB-SLAM2 and other state-of-the-art methods, particularly in dynamic scenes. For example, in static scenes, the mean RMSE is reduced to 0.050 m, a substantial improvement. In dynamic scenes, the system achieves a mean RMSE of 0.016 m when using RGB-D input, outperforming many existing methods.
  • Ablation Studies: Clearly show the positive impact of each proposed module: feature fusion, semantic consistency, and object-level BA. Removing any of these modules leads to a noticeable decrease in accuracy, with object-level BA and semantic consistency showing the most significant contributions to localization performance.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while JSDNet achieves real-time performance, the accuracy of both branches (depth and segmentation) still has room for improvement.
  • The proposed system relies on a fixed depth range for discretization, which might limit performance in scenes with extreme depth variations beyond the defined bounds.
  • Future Work:
    • The authors suggest investigating methods that combine semantics with multi-source data, such as RGB, depth, and inertial measurement units (IMUs), to further enhance localization robustness.
  • Exploring more advanced deep learning architectures and training strategies for JSDNet to push the boundaries of accuracy.

Summary updated

Abstract

SLAM is regarded as a fundamental task in mobile robots and AR, implementing localization and mapping in certain circumstances. However, with only RGB images as input, monocular SLAM systems suffer problems of scale ambiguity and tracking difficulty in dynamic scenes. Moreover, high-level semantic information can always contribute to the SLAM process due to its similarity to human vision. Addressing these problems, we propose a monocular object-level SLAM system enhanced by real-time joint depth estimation and semantic segmentation. The multi-task network, called JSDNet, is designed to predict depth and semantic segmentation simultaneously, with four contributions that include depth discretization, feature fusion, a weight-learned loss function, and semantic consistency optimization. Specifically, feature fusion facilitates the sharing of features between the two tasks, while semantic consistency aims to guarantee the semantic segmentation and depth consistency among various views. Based on the results of JSDNet, we design an object-level system that combines both pixel-level and object-level semantics with traditional tracking, mapping, and optimization processes. In addition, a scale recovery process is also integrated into the system to evaluate the truth scale. Experimental results on NYU depth v2 demonstrate state-of-the-art depth estimation and considerable segmentation precision under real-time performance, while the trajectory accuracy on TUM RGB-D shows less errors compared with other SLAM systems.
BibTeX
@inproceedings{gao2025monocular,
  author = {Ruicheng Gao and Yue Qi},
  title = {{Monocular Object-Level SLAM Enhanced by Joint Semantic Segmentation and Depth Estimation}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25072110},
}
Figure