VSLAM method based on object detection in dynamic environments

Jia Liu, Qiyao Gu, Dapeng Chen, Dong YanFrontiers Neurorobotics 2022 · DOI

TL;DR

This paper introduces a VSLAM (Visual Simultaneous Localization and Mapping) method designed for dynamic environments, a significant challenge for traditional VSLAM systems. By integrating object detection with YOLOv3 and a Gaussian Mixture Model (GMM) for background modeling, their approach effectively identifies and filters out dynamic objects, leading to more robust and accurate camera pose estimation and map building.

What Problem Does This Paper Address?

Augmented Reality (AR) applications increasingly demand VSLAM systems that can operate reliably in real-world environments, which are inherently dynamic. Traditional VSLAM algorithms often struggle in these scenarios, leading to excessive pose estimation errors and tracking failures due to the presence of moving objects like people and cars. These inaccuracies can cause virtual objects registered in AR to appear unstable or misplaced. The core problem this paper tackles is improving the robustness and accuracy of VSLAM in environments with significant motion, which is crucial for seamless AR experiences.

Methods

The proposed method builds upon the ORB-SLAM2 framework and enhances its tracking thread. Key innovations include:

  • Affine Transformation for Frame Correction: An affine transformation matrix is used to correct for shifts between consecutive frames, improving the stability of visual odometry.
  • GMM for Background Modeling and Dynamic Region Segmentation: A Gaussian Mixture Model (GMM) is employed to learn the background of the scene. This allows for the segmentation of foreground dynamic regions, providing prior information for object detection.
  • YOLOv3 for Dynamic Object Detection: The segmented dynamic regions are fed into a YOLOv3 detector to identify potential dynamic targets. YOLOv3 is chosen for its balance of accuracy and real-time performance, crucial for AR.
  • Improved Kalman Filter for Dynamic Object Tracking: An enhanced Kalman filter is used to predict and track the detected dynamic objects, ensuring continuity and robustness in tracking. This improved filter uses a nonlinear fitting curve based on previous observations to predict future states and a mechanism to select the most reliable observation.
  • Dynamic Feature Point Filtering: Before map building, feature points detected in the current frame are filtered to eliminate those belonging to dynamic objects, thus preventing them from corrupting the static map.

Datasets

The paper utilizes the TUM dataset for evaluating dynamic target detection and VSLAM performance. Specifically, they mention using the "Dynamic Object dataset in the TUM dataset" and various sequences from the fr2 and fr3 series (e.g., fr2/desk_with_person, fr3/sitting_static, fr3/w_xyz). They also use the MOT16 dataset for verifying the robustness of their multi-target tracking algorithm.

Key Insights and Ideas

  • GMM + YOLOv3 Synergy: The combination of GMM for background modeling and YOLOv3 for object detection is a powerful approach. GMM provides a robust way to identify dynamic areas, while YOLOv3 offers accurate and fast detection within those areas, creating a complementary system.
  • Improved Kalman Filter for Robust Tracking: The enhancement of the Kalman filter to handle noisy observations and select the most reliable data is critical for maintaining accurate tracking of dynamic objects, even with camera motion and lighting variations.
  • Dynamic Object Removal is Key for AR: The paper strongly emphasizes that accurately identifying and removing dynamic elements from the VSLAM pipeline is essential for stable and accurate AR registration. Static maps are fundamental for reliable virtual object placement.
  • Trade-off between YOLOv3 and YOLOv4: The authors acknowledge that while YOLOv4 is more efficient, they opted for YOLOv3 due to computational power constraints, highlighting a practical consideration in real-time systems.

Key Results

  • Improved Target Detection Accuracy: Table 1 shows that their method (Ours) achieves higher average Intersection over Union (IOU) values compared to traditional frame difference and optical flow methods, indicating more accurate detection of dynamic regions.
  • Robust VSLAM Performance: Figures 13 and 14, along with Tables 2-4, demonstrate that their proposed method significantly reduces Absolute Trajectory Error (ATE) and Rotational Drift (RPE) in dynamic environments compared to ORB-SLAM2, MR-SLAM, and SMR-SLAM. For instance, in some datasets, their RMSE values are considerably lower.
  • Stable AR Registration: Figures 15 and 16 showcase the improved stability of virtual object registration in dynamic scenarios. Their method maintains better alignment of virtual objects compared to ORB-SLAM2.
  • Real-time Performance: Table 5 indicates that their method achieves real-time performance, with registration latency generally below 25 ms, making it suitable for AR applications. While slightly slower than ORB-SLAM2 in some cases, it offers superior robustness.

Limitations and Future Work

  • Computational Cost: The authors acknowledge that while YOLOv3 is efficient, they did not use more advanced models like YOLOv4 due to computational power limitations. Future work will focus on optimizing computational cost for better real-time performance.
  • Performance in Highly Dynamic Scenes: While the method performs well in dynamic environments, the paper notes that performance might be slightly inferior to some advanced algorithms in low-dynamic datasets, though it still maintains an advantage over simpler methods.
  • Future YOLO Versions: The authors express interest in exploring newer YOLO versions (YOLOv5, YOLOv7) for potential improvements.
  • Further Optimization: Future work will aim to optimize computational cost to achieve even better real-time performance for Augmented Reality Registration.

Summary updated

BibTeX
@article{liu2022vslam,
  author = {Jia Liu and Qiyao Gu and Dapeng Chen and Dong Yan},
  title = {{VSLAM method based on object detection in dynamic environments}},
  journal = {Frontiers Neurorobotics},
  year = {2022},
  doi = {10.3389/fnbot.2022.990453},
}
Figure