Fusing Semantics and Motion State Detection for Robust Visual SLAM

Gaurav Singh, Meiqing Wu, Siew-Kei Lam

The Challenge of Dynamic Environments

Most visual SLAM (vSLAM) systems operate under a "static world" assumption, presupposing that the majority of the environment remains stationary. While outlier rejection schemes like RANSAC can handle minor movements, they often fail in highly dynamic settings such as crowded city streets or busy indoor hallways. In these scenarios, moving objects like pedestrians and vehicles can dominate the camera's field of view, leading to significant tracking drift or system failure.

Existing solutions generally fall into two categories: geometric methods and semantic methods. Geometric approaches detect motion via optical flow but are frequently noisy and struggle with precise object boundaries. Conversely, semantic methods use deep learning to mask out all potentially dynamic objects (e.g., every car in a frame). However, this often leads to "over-removal." For instance, a parked car is an excellent static landmark for SLAM; discarding it removes valuable features necessary for navigation, particularly in parking lots or residential areas.

A Multi-Stage Fusion Pipeline

To address these limitations, this research proposes a robust vSLAM system that fuses semantic information with geometric motion cues. The system introduces a three-stage preprocessing pipeline designed to integrate seamlessly with existing frameworks like ORB-SLAM2.

The proposed visual SLAM architecture featuring the three-stage moving feature removal pipeline.

1. Sparse Scene Flow Clustering

The system first calculates the 3D motion (scene flow) of sparse feature points. Using Delaunay triangulation, it constructs a graph of these points and prunes edges based on motion similarity and spatial proximity. This process effectively clusters points moving together as a single rigid body.

2. Semantic Segmentation

In parallel, a Convolutional Neural Network (SegNet) provides pixel-wise labels, identifying regions belonging to "movable" categories such as vehicles and people.

3. Probabilistic Fusion

The core innovation lies in the probabilistic fusion of these two streams. Rather than using simple binary logic, the authors employ Distance Transforms and Gaussian modeling to create a probability map.

A distance transform visualization used to create soft boundaries for probabilistic fusion.

By converting binary masks into distance maps, the system creates a "soft" boundary that accounts for sensor and segmentation uncertainty. This allows the system to distinguish between "movable" objects (semantics) and "actually moving" objects (geometry). Consequently, the system can retain features on a stationary car even if it belongs to a dynamic semantic class.

Experimental Evaluation

The proposed method was evaluated across three major benchmarks: TUM RGB-D (indoor), KITTI (outdoor automotive), and ApolloScape (highly dynamic urban scenes).

  • TUM Dataset: In high-dynamic "walking" sequences, the method reduced the Absolute Trajectory Error (ATE) by up to 90% compared to the baseline ORB-SLAM2.
  • ApolloScape: On challenging urban sequences, translation error was reduced from 33.03% to 22.44%. Notably, semantic-only methods performed worse in some cases because they removed too many static features, validating the fusion approach.
  • Ablation Studies: Results consistently showed that the combination of baseline, geometry, and semantics (B+G+S) outperformed using any single cue alone.

Key Insights and Future Directions

The research highlights that "movable" and "moving" are distinct states. A robust SLAM system must differentiate between them to maintain sufficient tracking features. By utilizing sparse scene flow and parallel processing for semantics, the system maintains near real-time efficiency.

Despite its success, the system remains dependent on the accuracy of the underlying semantic model; mislabeling static background elements as vehicles can still degrade performance. Future work may involve integrating more advanced real-time segmentation models, such as transformers, to further refine masks and improve robustness in increasingly chaotic environments.

Summary updated

Abstract

Achieving robust pose tracking and mapping in highly dynamic environments is a major challenge faced by existing visual SLAM (vSLAM) systems. In this paper, we increase the robustness of existing vSLAM by accurately removing moving objects from the scene so that they will not contribute to pose estimation and mapping. Specifically, semantic information is fused with motion states of the scene via a probability framework to enable accurate and robust moving object extraction in order to retain the useful features for pose estimation and mapping. Our work highlights the importance of distinguishing between motion states of potential moving objects for vSLAM in highly dynamic environments. The proposed method can be integrated into existing vSLAM systems to increase their robustness in dynamic environments without incurring much computation cost. We provide extensive experimental results on three well-known datasets to show that the proposed technique outperforms existing vSLAM methods in indoor and outdoor environments, under various scenarios such as crowded scenes.
BibTeX
@inproceedings{singh2020fusing,
  author = {Gaurav Singh and Meiqing Wu and Siew-Kei Lam},
  title = {{Fusing Semantics and Motion State Detection for Robust Visual SLAM}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093359},
}
Figure