A New Volumetric Fusion Strategy with Adaptive Weight Field for RGB-D Reconstruction

Xinqi Liu, Jituo Li, Guodong LuSensors 2020 · DOI

TL;DR

This paper introduces a novel volumetric fusion strategy for RGB-D reconstruction that significantly improves both geometry and texture quality. By employing an adaptive weight field to assess data reliability and a new texture fusion approach combining replacement, integration, and fixedness, the method effectively reduces texture blur, avoids incorrect textures, and suppresses noise, achieving state-of-the-art results comparable to offline methods in real-time.

What Problem Does This Paper Address?

High-quality 3D reconstruction is crucial for applications like autonomous driving, virtual reality, and robotics. However, existing real-time RGB-D reconstruction methods, particularly those based on volumetric fusion, struggle with texture blur and inaccuracies. Classical weighted averaging for texture fusion tends to smooth out fine details and introduce blur because it doesn't adequately account for slight misalignments in camera poses between frames, leading to averaging of color differences. While some methods focus on improving input data quality or using post-processing optimization, these often come with high computational costs or require good initial reconstructions, making them unsuitable for real-time applications. This paper aims to address the fundamental issue of texture blur and geometric noise in real-time volumetric fusion.

Methods

The proposed solution is a "New Volumetric Fusion Strategy" that acts as a module for existing reconstruction frameworks. It tackles the problem from two main angles:

  • Adaptive Weight Field: This is a core innovation. Instead of relying on simple heuristics or viewpoint-dependent weights, the authors develop a probabilistic and heuristic method to create an adaptive weight field. This field evaluates the reliability of RGB-D data by considering:
  • Photometric Consistency: Measures how consistent texture features are across frames, preserving local texture structure.
  • Edge Error: Identifies areas prone to texture errors (e.g., near occlusions) and reduces their influence.
  • Depth Edge Noise: Specifically models and suppresses noise in the depth map, particularly at edges, to improve geometric accuracy.
  • Viewpoint Dependence: Incorporates viewpoint information to control texture update ranges. This adaptive field guides the voxel fusion process, prioritizing high-quality data and mitigating the impact of noise and errors.

  • New Texture Fusion Strategy: This strategy combines three operations:

  • Replacement: When current frame data is significantly more reliable, it directly replaces previous texture information.
  • Integration: When current and previous frame reliability is similar, data is blended to maintain texture continuity and avoid seams.
  • Fixedness: When current frame reliability drops, the existing high-quality texture is preserved to prevent degradation from noisy data. This approach aims to preserve texture structure and reduce blur by intelligently deciding how to update voxel textures based on their assessed reliability.

Additionally, the paper proposes a two-step refined camera pose estimation method. It first uses frame-to-frame ICP for an initial pose and then refines it using a frame-to-model ICP that incorporates both geometric and photometric consistency, guided by the adaptive weight field. This improves pose accuracy, which is critical for high-quality texture reconstruction.

Datasets

The paper evaluates its method on several public RGB-D datasets, including:

  • Gate dataset
  • Hieroglyphics dataset
  • Tomb Statuary dataset
  • Lion dataset
  • Bricks dataset
  • Lucy dataset
  • Augustus dataset
  • Socrates dataset

For quantitative analysis, they also generate synthetic RGB-D data by adding varying levels of Gaussian noise to the Sokrates model and introducing slight camera pose perturbations. The paper does not introduce new benchmark datasets.

Key Insights and Ideas

  • Texture blur is not just a geometric alignment problem: While pose errors contribute, the core issue in texture fusion is how weighted averaging inherently smooths out details when pixel correspondences aren't perfect.
  • Data reliability is multi-faceted: Simply trusting depth data or assuming color is always good is insufficient. The paper's adaptive weight field elegantly combines photometric consistency, edge error, and depth noise to provide a more holistic reliability assessment.
  • Edge regions are critical and problematic: The paper highlights that edges are where texture errors are most likely to occur and where geometric noise can be most disruptive. Their edge error weight field specifically targets these areas.
  • A hybrid texture fusion strategy is key: No single operation (replacement, integration, fixedness) is sufficient. Combining them based on data reliability allows for both preserving fine details and maintaining continuity.
  • Pose refinement benefits from texture: Integrating photometric consistency into pose refinement, guided by the adaptive weight field, leads to more accurate camera poses, which in turn improves texture reconstruction.

Key Results

  • Qualitative Improvements: The method consistently produces sharper, clearer textures with better geometric detail compared to classical KinectFusion and even approaches the quality of offline methods like Intrinsic3D. Figures 2, 6, 7, and 8 visually demonstrate these improvements across various datasets.
  • Quantitative Accuracy:
    • Geometry: The method shows significant improvements in geometric accuracy, reducing errors by 15-25% for 1.0 mm voxel size and 22% for 2.0 mm voxel size compared to KinectFusion under varying noise levels (Table 2).
  • Texture: Texture accuracy is improved by approximately 20% for both 1.0 mm and 2.0 mm voxel sizes compared to KinectFusion (Table 3).
  • Noise Suppression: The adaptive weight field effectively suppresses noise, particularly in regions with large surface fluctuations, leading to smoother and more complete surfaces.
  • Pose Estimation: The two-step refined pose estimation method (TSW-ICP) significantly reduces accumulated pose error compared to standard ICP methods, with the adaptive weight field further enhancing accuracy.
  • Real-time Performance: The method is designed to be embedded as a module, aiming for real-time performance comparable to existing online methods.

Limitations and Future Work

  • Lighting Illumination: The current adaptive weighting field does not explicitly consider lighting illumination changes. This limits robustness in scenes with significant illumination variations. The authors suggest exploring simplified spherical harmonic illumination models for future work.
  • Specular Materials: The method is primarily applied to Lambertian materials. Reconstructing scenes with specular reflective materials may lead to significant errors.
  • Texture Discontinuity: While the TSW-ICP method helps, texture discontinuity can still occur with large camera pose evaluation errors.
  • Future Work: The authors plan to investigate efficient environmental illumination assessment models and extend the method to non-rigid scene reconstruction.

Summary updated

Abstract

High-quality 3D reconstruction results are very important in many application fields. However, current texture generation methods based on point sampling and fusion often produce blur. To solve this problem, we propose a new volumetric fusion strategy which can be embedded in the current online and offline reconstruction framework as a basic module to achieve excellent geometry and texture effects. The improvement comes from two aspects. Firstly, we establish an adaptive weight field to evaluate and adjust the reliability of data from RGB-D images by using a probabilistic and heuristic method. By using this adaptive weight field to guide the voxel fusion process, we can effectively preserve the local texture structure of the mesh, avoid wrong texture problems and suppress the influence of outlier noise on the geometric surface. Secondly, we use a new texture fusion strategy that combines replacement, integration, and fixedness operations to fuse and update voxel texture to reduce blur. Experimental results demonstrate that compared with the classical KinectFusion, our approach can significantly improve the accuracy in geometry and texture clarity, and can achieve equivalent texture reconstruction effects in real-time as the offline reconstruction methods such as intrinsic3d, even better in relief scenes.
BibTeX
@inproceedings{liu2020new,
  author = {Xinqi Liu and Jituo Li and Guodong Lu},
  title = {{A New Volumetric Fusion Strategy with Adaptive Weight Field for RGB-D Reconstruction}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20154330},
}
Figure