Dynamic Non-Rigid Objects Reconstruction with a Single RGB-D Sensor

Sen Wang, Xinxin Zuo, Chao Du, Runxiao Wang, Jiangbin Zheng, Ruigang Yang

TL;DR

This paper presents a novel framework for reconstructing dynamic, non-rigid 3D objects using a single RGB-D sensor. By introducing a global non-rigid registration approach with explicit loop closure and a model update step, they effectively tackle the error accumulation and drifting problems common in sequential fusion methods, achieving complete and watertight 3D models with high-fidelity color.

What Problem Does This Paper Address?

Reconstructing 3D models of objects that deform over time (non-rigid objects) using a single RGB-D sensor is a significant challenge. Traditional methods often struggle with accumulating errors from sequential frame-to-frame fusion, leading to "drifting" where the model becomes increasingly inaccurate over time. Furthermore, tracking surface deformations can be difficult, and existing approaches often rely on pre-scanned models or templates, limiting their applicability. This paper aims to overcome these limitations by developing a robust framework that can reconstruct dynamic non-rigid objects from scratch, without requiring prior models or strict pose constraints, and crucially, by addressing the drift problem through loop closure.

Methods

The proposed framework consists of several key stages:

  • Partial Pieces Generation: The input RGB-D sequence is divided into segments. For each segment, a "free form dynamic fusion" method is used to generate a partial 3D model (fragment) that captures geometric details and color.
  • Pairwise Non-Rigid Registration: Neighboring partial pieces are aligned using a non-rigid registration approach. This method employs an "Embedded Deformation Model" where graph nodes on a mesh are assigned affine transformations. The optimization minimizes a combination of as-rigid-as-possible, smoothness, and correspondence terms. Crucially, it leverages both geometric cues (from previously deformed meshes) and color information to resolve ambiguities.
  • Loop Detection: To combat drift, the system explicitly detects when a partial piece significantly overlaps with an earlier piece, forming a "loop." This is achieved by analyzing SIFT features extracted from the partial scans.
  • Global Non-Rigid Registration: Once loops are detected, a global registration framework is used to align all partial pieces simultaneously. This process incorporates correspondences from both neighboring pieces and those forming loops. The optimization uses a similar objective function to pairwise registration but applied globally.
  • Model Update: After global registration, a "model update" step refines the reconstructed model. This involves using the globally registered model as a proxy to iteratively update and align the individual pieces, further correcting misalignments and improving both geometry and appearance.

Datasets

The paper evaluates its approach on:

  • Synthetic Dataset: A textured mesh model was used to synthesize depth and color image sequences by moving a virtual camera. This allows for quantitative evaluation against a ground truth model.
  • Real Datasets: Several sequences of human subjects captured with a Microsoft Kinect V2 sensor were used for qualitative evaluation.

Key Insights and Ideas

  • Loop Closure is Crucial for Drift Mitigation: The paper strongly emphasizes that explicit loop detection and closure are essential to prevent the accumulation of errors that plague purely sequential fusion methods.
  • Combining Geometry and Appearance for Robust Registration: Relying solely on geometric information for non-rigid registration can be ambiguous. The authors demonstrate that incorporating color information significantly improves the accuracy and robustness of the alignment process.
  • Global Optimization for Consistent Reconstruction: Moving from pairwise to global registration, guided by loop closures, allows for a more consistent and globally optimal alignment of all partial scans, leading to a complete and watertight model.
  • Iterative Model Update for Refinement: The final model update step, using a proxy model, is key to addressing subtle misalignments that may persist after global registration, particularly in regions affected by occlusion.

Key Results

  • Quantitative Evaluation: On a synthetic dataset, the method achieved a mean error of 0.0023 m, with the largest errors (around 0.0041 m) occurring in thin structures like arms and hands.
  • Qualitative Evaluation: Experiments on real RGB-D sequences demonstrated the ability to reconstruct complete and watertight 3D models of dynamic non-rigid objects. The results showed significant improvements over existing methods like "3D self-portrait" and "VolumeDeform" in terms of alignment and completeness, particularly in handling occlusions and large deformations.
  • High-Fidelity Color Maps: The recovered models exhibit clear and sharp color maps, with textures well-preserved, thanks to the integration of appearance constraints during registration.
  • Application to Model Completion: The reconstructed models can be used for model completion tasks, where a partial input depth frame can be used to align and complete a previously reconstructed model.

Limitations and Future Work

  • Topology: The current method assumes a constant topology throughout the sequence, which might be a limitation for objects with significant topological changes.
  • Fast Motion: The partial piece generation might fail in cases of very fast motion.
  • Textureless Regions: While color information is used, the reliance on SIFT features for loop detection might still be an issue in highly textureless environments.

Future work suggested by the authors includes:

  • Texture Representation: Replacing per-vertex color with full textures for higher quality mesh appearance.
  • Learning-Based Correspondence: Investigating learning-based approaches for finding correspondences to handle fast motion and potentially improve robustness in textureless regions.
  • Changing Topology: Extending the framework to handle dynamic changes in object topology.
  • Applications: Developing applications like model-based view synthesis.

Summary updated

BibTeX
@inproceedings{wang2018dynamic,
  author = {Sen Wang and Xinxin Zuo and Chao Du and Runxiao Wang and Jiangbin Zheng and Ruigang Yang},
  title = {{Dynamic Non-Rigid Objects Reconstruction with a Single RGB-D Sensor}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=ST2O-oLY5Ig},
}
Figure