SplitFusion: Simultaneous Tracking and Mapping for Non-Rigid Scenes

Yang Li, Tianwei Zhang, Yoshihiko Nakamura, Tatsuya Harada

Abstract

We present SplitFusion, a novel dense RGB-D SLAM framework that simultaneously performs tracking and dense reconstruction for both rigid and non-rigid components of the scene. SplitFusion first adopts deep learning based semantic instant segmentation technique to split the scene into rigid or non-rigid surfaces. The split surfaces are independently tracked via rigid or non-rigid ICP and reconstructed through incremental depth map fusion. Experimental results show that the proposed approach can provide not only accurate environment maps but also well-reconstructed non-rigid targets, e.g. the moving humans.

Introduction

Traditional Visual SLAM (Simultaneous Localization and Mapping) systems typically rely on a "static world assumption." In these frameworks, any moving object is treated as an outlier or noise to be filtered out. While effective in controlled environments, this approach fails in real-world scenarios where dynamic elements—such as people walking through a room—are common. Such movement often leads to tracking failure or significant artifacts in the resulting 3D map.

While some "Dynamic SLAM" systems can handle moving rigid objects, non-rigid scenes involving deforming bodies or clothing remain a significant challenge. Existing solutions like DynamicFusion attempt to model the entire scene as a single deforming entity, which struggles with scalability and topological changes, such as a person moving behind furniture. SplitFusion addresses these issues by explicitly decomposing the scene into independent rigid and non-rigid components.

Methodology

The core of SplitFusion is a "divide and conquer" strategy that processes the scene as a collection of independent surfaces. The pipeline consists of four primary stages:

Overview of the SplitFusion pipeline, showing scene decomposition, canonical model generation, and volumetric integration.
  1. Scene Decomposition: The system processes RGB-D frames using YOLACT, a real-time instance segmentation network, to identify object categories. These are classified as "rigid" (e.g., furniture, floors) or "non-rigid" (e.g., humans). A 3D Graph Cut algorithm refines these segments using depth data to correct noisy edges from the neural network.
  2. Per-Surface Warping Field: Each surface is assigned a specific motion model. Rigid surfaces use 6-DoF transforms, while non-rigid surfaces utilize a deformation graph—a sparse set of nodes controlling local space warping.
  3. Independent Tracking: Tracking is performed independently for each surface. Rigid components use standard Iterative Closest Point (ICP), while non-rigid components undergo non-linear optimization minimizing a data term against the depth map and an As-Rigid-As-Possible (ARAP) regularization term.
  4. Surface Fusion: SplitFusion maintains separate Truncated Signed Distance Function (TSDF) volumes for each sub-surface. These are updated incrementally and unified in the camera's coordinate system for final visualization.

Evaluation and Key Results

The authors evaluated SplitFusion across several challenging datasets, including TUM RGB-D (fr3/walking_xyz), StaticFusion (Selfie), and VolumeDeform (Minion). These tests covered scenarios with moving people, camera-holding subjects, and non-rigid object interactions.

Reconstruction results at time steps t=1, 20, and 100, showing the evolution of both the static environment and dynamic human models.

Performance Metrics

On the TUM walking_xyz sequence, SplitFusion achieved an Absolute Trajectory Error (ATE) of 0.048m, significantly outperforming Co-Fusion (0.71m) and FlowFusion (0.12m). The system demonstrated remarkable robustness in the "Selfie" test, maintaining stable tracking even when a moving person dominated over 50% of the frame.

Qualitative Reconstruction

As shown in the temporal sequence of reconstructions, SplitFusion produces sharp, artifact-free maps of the static environment while simultaneously generating detailed 3D meshes of moving subjects. By treating surfaces independently, the system avoids the "streaking" artifacts common in global deformation models when objects move past one another.

Key Insights

  • Semantic Priors: Using YOLACT allows the system to immediately initialize the correct tracking pipeline based on object labels (e.g., "human") rather than waiting to observe motion.
  • Scalability: Splitting the scene into sub-surfaces avoids the quadratic computational growth associated with global non-rigid solvers, making the optimization more stable and efficient.
  • Handling Topology: Independent surface management allows the system to handle complex interactions, such as objects appearing or disappearing, without compromising the integrity of the global map.

Limitations and Future Work

While SplitFusion represents a significant step forward, it faces challenges with high memory consumption due to multiple TSDF volumes. Future iterations may explore surfel-based representations to reduce the memory footprint. Additionally, integrating deep-learning-based trackers could improve robustness to the rapid movements that occasionally challenge traditional ICP solvers.

Summary updated

Abstract

We present SplitFusion, a novel dense RGB-D SLAM framework that simultaneously performs tracking and dense reconstruction for both rigid and non-rigid components of the scene. SplitFusion first adopts deep learning based semantic instant segmentation technique to split the scene into rigid or non-rigid surfaces. The split surfaces are independently tracked via rigid or non-rigid ICP and reconstructed through incremental depth map fusion. Experimental results show that the proposed approach can provide not only accurate environment maps but also well-reconstructed non-rigid targets, e.g. the moving humans.
BibTeX
@inproceedings{li2020splitfusion,
  author = {Yang Li and Tianwei Zhang and Yoshihiko Nakamura and Tatsuya Harada},
  title = {{SplitFusion: Simultaneous Tracking and Mapping for Non-Rigid Scenes}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=4gXion5GHa1},
}
Figure