FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow

Tianwei Zhang, Huayan Zhang, Yang Li, Yoshihiko Nakamura, Lei Zhang

TL;DR

This paper introduces FlowFusion, a novel dense RGB-D SLAM system that tackles the challenge of dynamic environments by leveraging optical flow residuals for accurate dynamic/static segmentation. This allows for robust camera ego-motion estimation and precise reconstruction of static backgrounds, outperforming existing methods in complex, moving scenes.

What Problem Does This Paper Address?

Simultaneous Localization and Mapping (SLAM) systems are crucial for robots to navigate and build maps of their surroundings. However, a significant challenge arises in dynamic environments where moving objects (people, other robots, etc.) can disrupt the process. Traditional SLAM methods often assume a static world, meaning moving objects can lead to incorrect feature associations, inaccurate camera motion estimation, and ultimately, failed mapping. While some approaches try to filter out dynamic objects using semantic segmentation or object detection, they can struggle with unknown dynamic objects. This paper aims to address this by developing a SLAM system that can effectively identify and segment dynamic elements while simultaneously estimating camera motion and reconstructing the static environment.

Methods

FlowFusion's core innovation lies in its use of optical flow residuals to distinguish between static and dynamic elements within RGB-D data. Here's a breakdown of their approach:

  • Visual Odometry (VO) Front-end: The system starts with a robust VO front-end that optimizes for both photometric (color) and geometric (depth) alignment errors between consecutive RGB-D frames. This is a standard approach in dense SLAM, but FlowFusion enhances it by incorporating dynamic information.
  • Optical Flow Estimation: The RGB images from consecutive frames are fed into a PWC-net, a state-of-the-art deep learning model, to estimate dense optical flow. This flow represents the pixel-wise motion between frames.
  • Camera Ego-Motion Estimation: Simultaneously, the system estimates the camera's own motion (ego-motion) using the intensity and depth data.
  • Projected Scene Flow: The key novelty is the calculation of projected 2D scene flow. This is achieved by subtracting the estimated camera ego-motion flow from the observed optical flow. For static pixels, this projected scene flow should be close to zero. For dynamic pixels, it will be non-zero, with its magnitude indicating the speed of motion.
  • Dynamic/Static Segmentation: The projected scene flow residuals are then used to segment the scene. The point clouds are first clustered into supervoxels. The average optical flow residual for each cluster is computed. Clusters with high residuals are classified as dynamic, while those with low residuals are considered static. This segmentation is further refined by formulating an energy function that considers both individual cluster residuals and the connectivity between adjacent clusters, encouraging smooth segmentation.
  • Dynamic-Aware Reconstruction: Finally, the VO optimization (Equation 16) is modified to only consider static pixels for reconstruction. This ensures that moving objects do not corrupt the static map.

Datasets

The paper evaluates FlowFusion on two public RGB-D SLAM datasets:

  • TUM RGB-D dataset: This dataset is widely used for evaluating SLAM systems and includes sequences with dynamic elements.
  • HRPSlam dataset: This dataset features sequences captured by a humanoid robot, presenting challenges like fast rotations and shaking.

The authors do not introduce any new benchmarks.

Key Insights and Ideas

  • Optical Flow Residuals as a Universal Dynamic Indicator: The paper demonstrates that optical flow residuals, after accounting for camera ego-motion, provide a powerful and generalizable cue for identifying dynamic elements, without relying on pre-trained object detectors.
  • Model-Free Dynamic Segmentation: Unlike methods that depend on specific object detectors (like human detectors), FlowFusion's approach is model-free, meaning it can detect and segment a wider variety of dynamic objects.
  • Joint Optimization for Robustness: By jointly estimating camera motion and performing dynamic segmentation, the system achieves greater robustness in challenging dynamic scenarios.
  • Dense Reconstruction in Dynamic Scenes: The ability to accurately reconstruct static backgrounds even in the presence of moving objects is a significant step towards more practical SLAM applications.

Key Results

  • Quantitative Performance: FlowFusion shows competitive results on the TUM and HRPSlam datasets. While the object detection-based PF method achieved slightly better Absolute Trajectory Error (ATE) on the fr3/walking xyz sequence (4.1 cm vs. 12 cm for FF), FlowFusion demonstrated superior performance compared to other model-free dynamic SLAM methods like JF and SF. Table II and III provide detailed ATE and Relative Pose Error (RPE) RMSE values, where FF often ranks favorably among model-free approaches.
  • Qualitative Performance: Figure 4 and 6 showcase the dynamic segmentation results. FlowFusion effectively segments dynamic objects (humans, moving robot parts) and produces clean reconstructions of static backgrounds, outperforming JF and SF in handling complex dynamic scenes.

Limitations and Future Work

  • Sensitivity to Slight Motions: The authors acknowledge that, like other model-free methods, FlowFusion can be non-sensitive to very slight motions.
  • Fast Motions and Shaking: Very fast motions or significant camera shaking can lead to inaccurate optical flow estimations, which in turn can affect dynamic segmentation. Figure 6 illustrates this challenge with the HRPSlam sequence.
  • Future Work: The paper doesn't explicitly detail future work, but the limitations suggest potential avenues for improvement, such as enhancing sensitivity to subtle movements and developing more robust optical flow estimation techniques for highly dynamic or shaky scenarios.

Summary updated

Abstract

Dynamic environments are challenging for visual SLAM since the moving objects occlude the static environment features and lead to wrong camera motion estimation. In this paper, we present a novel dense RGB-D SLAM solution that simultaneously accomplishes the dynamic/static segmentation and camera ego-motion estimation as well as the static background reconstructions. Our novelty is using optical flow residuals to highlight the dynamic semantics in the RGB-D point clouds and provide more accurate and efficient dynamic/static segmentation for camera tracking and background reconstruction. The dense reconstruction results on public datasets and real dynamic scenes indicate that the proposed approach achieved accurate and efficient performances in both dynamic and static environments compared to state-of-the-art approaches.
BibTeX
@inproceedings{zhang2020flowfusion,
  author = {Tianwei Zhang and Huayan Zhang and Yang Li and Yoshihiko Nakamura and Lei Zhang},
  title = {{FlowFusion: Dynamic Dense RGB-D SLAM Based on Optical Flow}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=qytCgeYNrQ},
}
Figure