A Unified Framework for Mutual Improvement of SLAM and Semantic Segmentation

Kai Wang, Yimin Lin, Luowei Wang, Liming Han, Minjie Hua, Xiang Wang, Shiguo Lian, Bill Huang

Here is a summary of the paper:

TL;DR

Visual SLAM and semantic segmentation are traditionally treated as separate computer vision tasks, but this paper introduces a unified framework where they actively improve each other. By using segmentation to mask out moving objects, the system achieves highly accurate SLAM tracking; simultaneously, it uses the 3D camera pose generated by SLAM to project past frames forward, fixing errors and refining boundaries in the 2D segmentation masks.

What Problem Does This Paper Address?

For a robot to navigate effectively, it needs to know where it is (Localization/SLAM) and what is around it (Semantic Segmentation). Both tasks face distinct challenges in the real world:

  • The Dynamic World Problem in SLAM: Standard visual SLAM algorithms assume the world is static. If a person walks through the frame, the algorithm might track features on the person, causing the camera's estimated trajectory to drift or fail entirely.
  • Temporal Inconsistency in Segmentation: 2D image-based deep learning segmentation models process frames individually. This often leads to flickering masks, imprecise boundaries, or objects being completely missed in one frame despite being detected in the previous one.

Previously, researchers tried using segmentation to help SLAM (by masking out cars or people), but they rarely used SLAM's geometric data to feed back into and improve the deep learning segmentation model.

Methods

The authors propose an interwoven framework using FCIS (for instance-aware semantic segmentation) and ORB-SLAM2 (for RGB-D tracking and mapping). The pipeline works in a continuous loop for each frame:

  1. Initial Segmentation & Coarse Pose: The incoming RGB frame is segmented to identify potentially "moveable" objects (e.g., people, cars, chairs). Simultaneously, the SLAM module calculates a coarse camera pose.
  2. Dynamic Feature Rejection: The system checks the ORB feature points falling inside the "moveable" segmentation masks. By comparing their movement against the coarse pose, it classifies them as either truly moving or currently static. Moving points are discarded.
  3. Fine Pose & Mapping: A highly accurate "fine pose" is calculated using only the reliable, static background points. The system updates two maps: a tracking map for immediate navigation, and a long-term map that strictly excludes any potentially moveable objects so future visits to the area remain accurate.
  4. Segmentation Refinement: Here is the novel twist. The system takes the refined 3D pose and uses it to project the segmentation masks from the previous frame into the current frame. By comparing the projected mask with the current frame's initial mask, it can recover missing parts (e.g., if the neural network failed to see a table due to motion blur) or shrink oversized, inaccurate boundaries.

Datasets

  • TUM Dataset: Used to evaluate vSLAM performance, specifically the 'fr3' sequences featuring people walking and sitting in an office.
  • ScanNet Dataset: Used to evaluate segmentation accuracy. It contains indoor RGB-D sequences with dense ground-truth annotations.
  • AirSim Generated Dataset: Because existing datasets lacked ground truth for both dynamic relocalization and segmentation simultaneously, the authors generated a custom synthetic dataset using Microsoft AirSim. It features 40 virtual city routes with moving vehicles and pedestrians.

Key Insights and Ideas

  • A Two-Way Street for Data: The most important takeaway is that geometric 3D data (SLAM) and semantic 2D data (CNN segmentation) are highly complementary. Geometry provides the temporal consistency that 2D CNNs lack, while semantics provide the context (what moves and what doesn't) that pure geometry struggles to deduce.
  • Moveable vs. Moving: The framework smartly distinguishes between objects that are moving and objects that could move. A parked car is static for the current tracking frame, but it shouldn't be added to the long-term map because it might be gone tomorrow.
  • Geometric Priors Fix Neural Network Hallucinations: By projecting the previous frame's mask using exact camera math, the system creates a geometric prior. If the CNN suddenly fails to segment a chair in frame 2, the projection from frame 1 acts as a reliable fallback.

Key Results

  • Massive SLAM Improvements: On the TUM Walking_halfsphere sequence, the framework achieved an Absolute Trajectory Error (ATE) of just 0.019m, crushing standard ORB-SLAM2 (0.351m) and slightly edging out the state-of-the-art dynamic SLAM system, DynaSLAM (0.025m).
  • Better Segmentation: On the ScanNet dataset, the SLAM-refined segmentation improved the mean Intersection over Union (mIoU) from 0.5620 (baseline FCIS) to 0.5751.
  • Relocalization Success: On the custom AirSim dataset, the dual-map strategy proved highly effective for revisiting areas. Relocalization ATE dropped from 0.82m (ORB-SLAM2) to 0.39m.
  • Speed: The system is near real-time. Initial segmentation takes ~113ms, refinement takes ~50ms, and fine tracking/mapping takes ~22ms per frame.

Limitations and Future Work

  • Fast Camera Motion: The segmentation refinement relies heavily on the assumption that there are no drastic changes between adjacent frames. If the camera moves too fast or the frame rate is too low, the region matching algorithm can fail and produce fake results.
  • Future Work: The authors suggest introducing frame interpolation to handle fast-moving objects and low frame rates. They also note that the 50ms segmentation refinement step could be significantly accelerated using parallel computing or GPU optimization, pushing the entire framework closer to strict real-time performance for autonomous driving and UAV applications.

Summary updated

Abstract

This paper presents a novel framework for simultaneously implementing localization and segmentation, which are two of the most important vision-based tasks for robotics. While the goals and techniques used for them were considered to be different previously, we show that by making use of the intermediate results of the two modules, their performance can be enhanced at the same time. Our framework is able to handle both the instantaneous motion and long-term changes of instances in localization with the help of the segmentation result, which also benefits from the refined 3D pose information. We conduct experiments on various datasets, and prove that our framework works effectively on improving the precision and robustness of the two tasks and outperforms existing localization and segmentation algorithms.
BibTeX
@inproceedings{wang2019unified,
  author = {Kai Wang and Yimin Lin and Luowei Wang and Liming Han and Minjie Hua and Xiang Wang and Shiguo Lian and Bill Huang},
  title = {{A Unified Framework for Mutual Improvement of SLAM and Semantic Segmentation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=DgrVGH7AAp},
}

Loading related content...

Figure