SplitFusion: Simultaneous Tracking and Mapping for Non-Rigid Scenes

Yang Li, Tianwei Zhang, Yoshihiko Nakamura, Tatsuya Harada

TL;DR

SplitFusion is a dense RGB-D SLAM framework that refuses to ignore the "messiness" of the real world. Unlike traditional SLAM that treats moving objects as noise to be discarded, SplitFusion uses semantic instance segmentation to split a scene into independent rigid and non-rigid surfaces, allowing it to simultaneously track the camera and reconstruct deforming objects like moving humans.

What Problem Does This Paper Address?

Most Visual SLAM (Simultaneous Localization and Mapping) systems operate on a "static world assumption." They assume that if something moves, it’s an outlier. In practice, this means dynamic elements—like a person walking through a room—are either masked out or cause the camera tracking to fail and the 3D map to become a blurry mess of artifacts.

While some "Dynamic SLAM" systems can handle moving rigid objects (like a sliding box), non-rigid scenes (deforming bodies, clothing, etc.) remain a massive challenge. Existing solutions like DynamicFusion attempt to model the whole scene as one giant deforming entity, but these don't scale well to large rooms and struggle when the "topology" of a scene changes—for example, when a person walks behind a desk or picks up an object.

Methods

The core philosophy of SplitFusion is "divide and conquer." The pipeline follows four main steps:

  1. Scene Decomposition: The system takes RGB-D frames and passes the color data through YOLACT, a real-time instance segmentation network. This identifies 80+ object categories. The system then classifies these as "rigid" (chairs, floors) or "non-rigid" (humans, pets). To fix the noisy edges typical of neural networks, they refine these segments using a 3D Graph Cut algorithm on the depth data.
  2. Per-Surface Warping Field: Each segmented surface is assigned its own motion model. Rigid surfaces are tracked using standard 6-DoF transforms. Non-rigid surfaces use a deformation graph—a sparse set of nodes that control how the surrounding space "warps" over time.
  3. Independent Tracking: The system performs tracking for each surface independently.
  4. Rigid surfaces use standard Iterative Closest Point (ICP).
  5. Non-rigid surfaces use a non-linear optimization that minimizes a "data term" (how well the model fits the new depth map) and an ARAP (As-Rigid-As-Possible) regularization term to ensure the deformation looks physically plausible.
  6. Surface Fusion: Instead of one global volume, SplitFusion maintains separate TSDF (Truncated Signed Distance Function) volumes for each sub-surface. These are updated incrementally and then re-united in the camera's coordinate system for the final visualization.

Datasets

The authors evaluated SplitFusion on three distinct types of sequences to test different challenges:

  • TUM RGB-D (fr3/walking_xyz): A standard benchmark featuring people walking in an office while the camera moves.
  • StaticFusion (Selfie): A challenging sequence where a person holds the camera while moving, meaning the "dynamic" object occupies more than 50% of the frame.
  • VolumeDeform (Minion): A sequence focusing on a person interacting with a deforming toy, testing high-entanglement non-rigid tracking.

Key Insights and Ideas

  • Semantics as a Prior: By using YOLACT, the system doesn't have to "guess" if an object is non-rigid based on motion alone. It knows a "human" label implies potential deformation, allowing it to initialize the correct tracking pipeline immediately.
  • Breaking the Scalability Barrier: By splitting the scene into sub-surfaces, the system avoids the quadratic computational growth seen in global non-rigid solvers. Solving several small optimization problems is often faster and more stable than solving one massive one.
  • Handling Topology Changes: Because surfaces are independent, SplitFusion can handle objects appearing, disappearing, or moving past each other without the "streaking" artifacts common in single-model deformation systems.

Key Results

  • Superior Accuracy: On the TUM walking_xyz sequence, SplitFusion achieved an Absolute Trajectory Error (ATE) of 0.048m, significantly outperforming Co-Fusion (0.71m) and FlowFusion (0.12m).
  • Clean Reconstructions: Qualitatively, the system produces sharp maps of the static environment (desks, floors) while simultaneously providing detailed, denoised 3D meshes of moving people.
  • Robustness: In the "Selfie" test, where the moving person dominates the view, SplitFusion maintained stable camera tracking where traditional methods typically drift or fail.

Limitations and Future Work

Despite its strengths, the authors note a few hurdles:

  • Computational Heavyweight: Maintaining multiple TSDF volumes is memory-intensive. The authors suggest moving toward a surfel-based representation (point-based) in the future to reduce the memory footprint.
  • Fast Motion: Like most ICP-based methods, very rapid movements can cause the tracking to lose alignment.
  • Deep Tracking Integration: The authors propose incorporating "deep trackers" (neural networks trained specifically for motion estimation) to make the system more robust to the large displacements that currently challenge the ICP solver.

Summary updated

Abstract

We present SplitFusion, a novel dense RGB-D SLAM framework that simultaneously performs tracking and dense reconstruction for both rigid and non-rigid components of the scene. SplitFusion first adopts deep learning based semantic instant segmentation technique to split the scene into rigid or non-rigid surfaces. The split surfaces are independently tracked via rigid or non-rigid ICP and reconstructed through incremental depth map fusion. Experimental results show that the proposed approach can provide not only accurate environment maps but also well-reconstructed non-rigid targets, e.g., the moving humans.
BibTeX
@inproceedings{li2020splitfusion,
  author = {Yang Li and Tianwei Zhang and Yoshihiko Nakamura and Tatsuya Harada},
  title = {{SplitFusion: Simultaneous Tracking and Mapping for Non-Rigid Scenes}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=4gXion5GHa1},
}

Loading related content...

Figure