HRBF-Fusion: Accurate 3D Reconstruction from RGB-D Data Using On-the-fly Implicits

Yabin Xu, Liangliang Nan, Laishui Zhou, Jun Wang, Charlie C. L. WangACM Trans. Graphics 2022 · DOI

TL;DR

This paper introduces HRBF-Fusion, a novel 3D reconstruction system that leverages Hermite Radial Basis Functions (HRBFs) for continuous surface representation. By using on-the-fly HRBFs, the system achieves more robust camera tracking and higher-fidelity 3D reconstructions from RGB-D data, overcoming limitations of traditional discrete surface representations.

What Problem Does This Paper Address?

Reconstructing high-fidelity 3D objects and scenes from RGB-D data is crucial for applications like augmented reality, digital fabrication, and robotics. Existing methods often struggle with accumulating errors in camera tracking and distortions in the reconstructed surfaces. These issues stem from the discrete nature and limited resolution of common surface representations (e.g., point clouds or voxels), which are susceptible to noise and can lead to smoothed-out details or artifacts, especially in high-curvature regions. Furthermore, camera tracking, a critical component, is prone to drift, particularly in featureless areas or when dealing with noisy depth data.

Methods

The core innovation of HRBF-Fusion lies in its use of on-the-fly Hermite Radial Basis Functions (HRBFs) as a continuous surface representation. Unlike discrete methods, HRBFs provide a smooth, implicit surface that can be evaluated efficiently at any point. This continuous representation is integrated into a standard RGB-D fusion framework, enhancing several key stages:

  • Surface Evaluation: HRBFs are used to evaluate continuous surfaces for both the input RGB-D frames and the global 3D model. This allows for more accurate prediction of surface points, normals, and importantly, curvature.
  • Robust Camera Tracking: The continuous HRBF surface enables more reliable curvature estimation. This information is then used to weight the camera pose estimation (registration) process, significantly improving robustness, especially in noisy or challenging environments. The system also derives a "reconstruction-indicated confidence map" from the HRBF surface properties, which helps to mitigate the impact of noise and outliers in depth data.
  • Depth Map Fusion: The improved camera tracking and confidence maps lead to more accurate fusion of new depth data into the global model.
  • Framework: The system builds upon existing RGB-D fusion pipelines, incorporating submap-based hierarchical optimization (similar to ElasticFusion and ORB-SLAM2) for handling large scenes and mitigating accumulated errors.

Datasets

The authors evaluated their method on a variety of datasets:

  • Object Scans [Choi et al. 2016]: A large dataset of individual 3D object scans.
  • ICL-NUIM benchmark [Handa et al. 2014]: A synthetic dataset with ground truth for geometric and camera poses, used for evaluating tracking accuracy and reconstruction quality.
  • TUM benchmark [Sturm et al. 2012]: Real-world RGB-D sequences with ground truth camera poses, used for evaluating tracking accuracy.
  • CoRBS benchmark [Wasenmüller et al. 2016]: A dataset with ground truth 3D models and camera poses, used for evaluating reconstruction accuracy.
  • CuFusion dataset [Zhang and Hu 2017]: Contains synthetic and real-world sequences.
  • ScanNet dataset [Dai et al. 2017]: A large dataset of RGB-D videos of indoor scenes.
  • Custom Datasets: The authors also captured their own data, including large indoor scenes and an urban street sequence, to demonstrate scalability and detail recovery.

Key Insights and Ideas

  • Continuous Representation is Key: The paper strongly advocates for the benefits of continuous surface representations (like HRBFs) over discrete ones for 3D reconstruction. This smoothness inherently handles noise better and provides richer geometric information.
  • Curvature for Robust Tracking: Leveraging accurate curvature estimation derived from HRBFs significantly improves camera tracking robustness, especially in challenging scenarios where traditional methods fail.
  • Confidence from Surface Properties: The novel "reconstruction-indicated confidence map" derived from HRBF surface properties is more effective than traditional distortion-based methods for identifying unreliable depth measurements.
  • On-the-Fly Efficiency: The "on-the-fly" nature of HRBF evaluation makes the system computationally feasible for near real-time applications, a critical requirement for interactive 3D reconstruction.

Key Results

  • Improved Tracking Robustness: HRBF-Fusion demonstrated significantly lower tracking errors compared to state-of-the-art methods (Kintinuous, PointFusion, VoxelHashing) under various noise levels. For instance, in the Lego-PAMI-TT benchmark, their method showed a dramatic improvement in registration accuracy.
  • Higher Reconstruction Accuracy: The system achieved better or comparable reconstruction accuracy on various benchmarks, including the ICL-NUIM and CoRBS datasets, often recovering finer geometric details than competing methods.
  • Scalability: The method successfully reconstructed large-scale scenes (over 21 meters) with high fidelity, outperforming other systems in handling complex trajectories and local loops.
  • Efficiency: The system achieves an average processing time of 42ms per frame (approximately 24Hz) on a single GPU, with HRBF prediction being the most time-consuming component.
  • Memory Efficiency: HRBF-Fusion showed a significantly smaller memory footprint compared to BundleFusion, especially when aiming for finer details.

Limitations and Future Work

  • Featureless Regions: The authors acknowledge that camera tracking can still fail between intra-submaps in featureless regions (e.g., white planar walls), a common problem in RGB-D reconstruction.
  • Future Work:
    • Investigating proactive reconstruction methods using robotic systems.
  • Incorporating geometric primitives or structural regularities (like parallelism or orthogonality) to enhance hierarchical optimization for long-range scanning.

Summary updated

BibTeX
@article{xu2022hrbf,
  author = {Yabin Xu and Liangliang Nan and Laishui Zhou and Jun Wang and Charlie C. L. Wang},
  title = {{HRBF-Fusion: Accurate 3D Reconstruction from RGB-D Data Using On-the-fly Implicits}},
  journal = {ACM Trans. Graphics},
  year = {2022},
  doi = {10.1145/3516521},
}
Figure