Dynamic 3D scenario-oriented monocular SLAM based on semantic probability prediction

Xiaokun Pan, Haomin Liu, Ming Fang, Zheng Wang, Yong Zhang, Guofeng ZhangJournal of Image and Graphics 2023 · DOI

Abstract

Visual Simultaneous Localization and Mapping (vSLAM) is a cornerstone of robotics and computer vision, enabling devices to perceive 3D environments and determine their own position. However, standard vSLAM systems often fail in real-world settings because they rely on a "static world assumption." When objects move, these systems suffer from tracking drift or failure. This paper introduces DSP-SLAM, a novel monocular visual SLAM algorithm that utilizes semantic probability prediction to handle dynamic scenes. By combining semantic segmentation with robust geometric estimation, the system represents the motion state of observations as probabilities. This allows it to filter out dynamic interference while retaining stable features from stationary objects, significantly improving tracking accuracy and map quality in complex environments.

Introduction

Simultaneous Localization and Mapping (SLAM) is the "holy grail" of spatial awareness for autonomous systems. Despite its success, the traditional reliance on a frozen, static world remains a major hurdle. In dynamic environments—such as a busy office or a crowded street—moving people or objects can confuse the system, leading to significant localization errors.

Existing solutions typically either use geometric outliers to identify motion (which fails in high-dynamic scenes) or use semantic labels to blindly delete all potentially moving objects (which discards valuable stable data). DSP-SLAM proposes a middle ground: a probabilistic approach that determines if a "person" or "chair" is actually moving before deciding whether to use its visual features for tracking.

Methodology

The DSP-SLAM system architecture, integrating semantic segmentation with probabilistic motion state estimation.

The DSP-SLAM pipeline operates through several integrated modules designed to handle dynamic uncertainty:

  1. Semantic Segmentation & Association: The system employs PanopticFCN to generate semantic labels for each frame. To maintain temporal consistency, these labeled clusters are tracked across frames using optical flow and spatial overlap, ensuring that the system understands the identity of objects over time.
  2. Motion State Detection: Rather than assuming all objects with certain labels (like "person") are moving, the system uses a modified RANSAC variant called PARSAC. It applies geometric constraints, specifically the Sampson error, to check if a cluster's movement aligns with the camera's ego-motion.
  3. Semantic Probability Prediction: This core innovation uses a Logistic regression model to calculate the probability that a specific 2D observation is dynamic. This "soft" logic allows the system to prioritize features that are statistically likely to be static.
  4. Weighted Pose Estimation: During camera pose estimation (the PnP problem), observations are weighted by their static probability. This ensures that even if a person is in the frame, if they are stationary, their features can still contribute to the system's stability.
  5. Map Maintenance: The system proactively culls "invalid" map points—those that have transitioned from static to dynamic—preventing the 3D map from being cluttered with "ghost" artifacts.

Experimental Evaluation

Example of semantic segmentation and mask generation on the custom dynamic dataset.

The researchers evaluated DSP-SLAM using two primary sources:

  • TUM-RGBD Dataset: The industry standard for SLAM evaluation, focusing on "walking" and "sitting" sequences with varying dynamic levels.
  • Custom Dynamic Dataset: A high-precision dataset captured using a VICON motion capture system to provide ground truth. This included challenging scenarios like moving objects on a desktop and crowded indoor environments.

Key Results

DSP-SLAM demonstrated a significant leap in performance over baselines like ORB-SLAM2 and dynamic-focused systems like DynaSLAM:

  • Accuracy: In the "Desktop 1" sequence, DSP-SLAM achieved an Absolute Trajectory Error (ATE) of 2.48 cm, a massive improvement over ORB-SLAM2's 15.70 cm.
  • Robustness: In high-dynamic "Walking" sequences, the system maintained a tracking percentage of over 91%, where standard SLAM systems often failed entirely.
  • AR Stability: Qualitative tests in Augmented Reality (AR) showed that virtual objects remained pinned to the real world even as people moved objects in the scene, highlighting the system's superior mapping quality.

Conclusion

By moving away from binary "static vs. dynamic" classifications and adopting a probabilistic semantic model, DSP-SLAM provides a robust solution for monocular SLAM in the real world. While current processing speeds (15 fps) present a minor bottleneck for real-time high-speed robotics, the system's ability to maintain high accuracy in crowded, changing environments marks a significant advancement for autonomous navigation and AR applications.

Summary updated

Abstract

目的基于视觉的同步定位与建图(visual-based simultaneous localization and mapping,vSLAM)是计算机视觉以及机器人领域中的关键技术,其通过对输入的图像进行处理分析来感知周围的3维环境以及进行自身的定位。现有的SLAM系统大多依赖静态世界假设,在真实环境中的动态物体会严重影响视觉SLAM系统的稳定运行。同时,场景中静止与运动部分往往和其语义有密切关系,因而可以借助场景中的语义信息来提升视觉SLAM系统在动态环境下的稳定性。为此,提出一种新的基于语义概率预测的面向动态场景的单目视觉SLAM算法。方法结合语义分割的结果以及鲁棒性估计算法,通过对分割进行数据关联、状态检测,从概率的角度来表示观测的静止/运动状态,剔除动态物体上的观测对相机位姿估计的干扰,同时借助运动概率及时剔除失效的地图点,使系统在复杂动态的场景中依然能够稳定运行。结果在本文构建的复杂动态场景数据集上,提出的方法在跟踪精度和完整度上都显著优于现有的单目视觉SLAM方法,而且在TUM-RGBD数据集中的多个高动态序列上也取得了更好的结果。此外,本文定性比较了动态场景下的建图质量以及AR(augmented reality)效果。结果表明,本文方法明显优于对比方法。结论本文通过结合语义分割信息以及鲁棒性估计算法,对分割区域进行数据关联以及运动状态检测,以概率的形式表示2D观测的运动状态,同时及时剔除失效地图点,使相机位姿估计的精度以及建图质量有了明显提升,有效提高了单目视觉SLAM在高度动态环境中运行的鲁棒性。
BibTeX
@article{pan2023dynamic,
  author = {Xiaokun Pan and Haomin Liu and Ming Fang and Zheng Wang and Yong Zhang and Guofeng Zhang},
  title = {{Dynamic 3D scenario-oriented monocular SLAM based on semantic probability prediction}},
  journal = {Journal of Image and Graphics},
  year = {2023},
  doi = {10.11834/jig.210632},
}
Figure