A Unified Framework for Mutual Improvement of SLAM and Semantic Segmentation

Kai Wang, Yimin Lin, Luowei Wang, Liming Han, Minjie Hua, Xiang Wang, Shiguo Lian, Bill Huang

Abstract

This paper presents a novel framework for simultaneously implementing localization and segmentation, which are two of the most important vision-based tasks for robotics. While the goals and techniques used for them were considered to be different previously, we show that by making use of the intermediate results of the two modules, their performance can be enhanced at the same time. Our framework is able to handle both the instantaneous motion and long-term changes of instances in localization with the help of the segmentation result, which also benefits from the refined 3D pose information. We conduct experiments on various datasets, and prove that our framework works effectively on improving the precision and robustness of the two tasks and outperforms existing localization and segmentation algorithms.

Introduction

For a robot to navigate effectively, it must solve two distinct but related problems: knowing its location (Localization/SLAM) and understanding its surroundings (Semantic Segmentation). Traditionally, these are treated as separate tasks. However, both face significant challenges in dynamic real-world environments. Standard visual SLAM algorithms often fail when dynamic objects, such as pedestrians, enter the frame and cause trajectory drift. Conversely, 2D deep learning segmentation models often suffer from temporal inconsistency, leading to flickering masks or missed detections across consecutive frames.

This paper proposes a unified framework that leverages the synergy between these tasks. By using segmentation to identify and mask out moving objects, the system achieves highly accurate SLAM tracking. Simultaneously, the refined 3D camera pose generated by SLAM is used to project past segmentation masks forward, correcting errors and refining boundaries in the current frame.

Methodology

The proposed framework integrates instance-aware semantic segmentation (FCIS) with RGB-D tracking and mapping (ORB-SLAM2). The system operates in a continuous loop for each frame, utilizing a two-way data exchange between geometric and semantic modules.

Figure 1: The proposed unified framework architecture illustrating the interaction between the Segmentation and vSLAM modules to achieve mutual improvement.
  1. Initial Processing: The system identifies potentially "moveable" objects (e.g., people, cars) and calculates a coarse camera pose.
  2. Dynamic Feature Rejection: Feature points within moveable masks are analyzed against the coarse pose to distinguish between truly moving objects and static ones. Moving points are discarded to ensure tracking stability.
  3. Fine Pose & Mapping: A highly accurate "fine pose" is calculated using only reliable static background points. The system maintains a tracking map for immediate navigation and a long-term map that excludes dynamic objects.
  4. Segmentation Refinement: The refined 3D pose is used to project previous segmentation masks into the current frame. This geometric prior allows the system to recover missing detections or correct imprecise boundaries produced by the neural network.

Experimental Results

The framework was evaluated using the TUM dataset for SLAM performance, the ScanNet dataset for segmentation accuracy, and a custom AirSim-generated dataset for dynamic relocalization.

SLAM and Relocalization Performance

On the TUM Walking_halfsphere sequence, the framework achieved an Absolute Trajectory Error (ATE) of just 0.019m, significantly outperforming standard ORB-SLAM2 (0.351m) and edging out the specialized DynaSLAM (0.025m). The dual-map strategy also improved relocalization on the AirSim dataset, dropping the ATE from 0.82m to 0.39m.

Figure 6: Comparison of estimated camera trajectories against ground truth. The proposed framework (green) maintains high accuracy in dynamic scenes where standard ORB-SLAM2 (red) suffers from significant drift.

Segmentation Accuracy

By incorporating geometric priors from the SLAM module, the segmentation accuracy on the ScanNet dataset improved from a baseline mean Intersection over Union (mIoU) of 0.5620 to 0.5751. This demonstrates that geometric 3D data provides the temporal consistency that 2D CNNs often lack.

Conclusion

This research demonstrates that geometric 3D data and semantic 2D data are highly complementary. Geometry provides temporal consistency for segmentation, while semantics provide the context necessary for robust SLAM in dynamic environments. While the system currently operates near real-time, future work will focus on optimizing the refinement steps and introducing frame interpolation to handle high-speed camera motion.

Summary updated

Abstract

This paper presents a novel framework for simultaneously implementing localization and segmentation, which are two of the most important vision-based tasks for robotics. While the goals and techniques used for them were considered to be different previously, we show that by making use of the intermediate results of the two modules, their performance can be enhanced at the same time. Our framework is able to handle both the instantaneous motion and long-term changes of instances in localization with the help of the segmentation result, which also benefits from the refined 3D pose information. We conduct experiments on various datasets, and prove that our framework works effectively on improving the precision and robustness of the two tasks and outperforms existing localization and segmentation algorithms.
BibTeX
@inproceedings{wang2019unified,
  author = {Kai Wang and Yimin Lin and Luowei Wang and Liming Han and Minjie Hua and Xiang Wang and Shiguo Lian and Bill Huang},
  title = {{A Unified Framework for Mutual Improvement of SLAM and Semantic Segmentation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=DgrVGH7AAp},
}
Figure