BundleRecon: Ray Bundle-Based 3D Neural Reconstruction

Weikun Zhang, Jianke ZhuarXiv 2023 · DOI · arXiv

Introduction

Neural implicit representations, such as NeRF, NeuS, and IDR, have revolutionized 3D reconstruction from 2D images. Despite their success, these methods typically rely on sampling individual, discrete rays. This approach captures longitudinal data along the ray but ignores transversal information—the relationship between a pixel and its neighbors. This "tunnel vision" often results in noisy geometry, missing details, or physical "holes" in the reconstructed mesh, particularly when pixel sampling is sparse.

Methodology

The authors introduce BundleRecon, a module designed to integrate into existing neural reconstruction pipelines. The core shift is from point-based sampling to patch-based sampling.

Ray Bundle Construction

Instead of sampling $n$ individual rays, BundleRecon samples $n$ patches of $s \times s$ pixels (e.g., $3 \times 3$ bundles). This is analogous to using a "flashlight" to illuminate a surface area rather than a "laser pen" to pinpoint a single dot.

The "flashlight" analogy: Ray bundles (bottom) capture local surface context compared to traditional single-ray "laser" sampling (top).

Bundle-Based Loss Functions

To leverage the spatial information within these patches, three new loss components are introduced:

  • Mean and Variance Loss ($L_m, L_v$): These ensure the statistical color distribution of the rendered patch aligns with the ground truth, acting as a regularizer for smoother surfaces.
  • Convolutional Loss ($L_{conv}$): By applying a Sobel kernel to the patches, the model extracts edge and texture features, providing structural signals that single-pixel methods lack.
  • Distance Masking ($M_P$): A spatial distance mask filters out irrelevant background information when a bundle overlaps an object boundary, preventing geometric confusion.

Experiments and Results

BundleRecon was evaluated on the DTU and BlendedMVS datasets. It demonstrated broad compatibility, functioning as a plug-in for both IDR (ray tracing) and NeuS (volume rendering).

Quantitative Performance

On the DTU dataset, BundleRecon improved the mean Chamfer distance—a key metric for 3D accuracy. When added to IDR, the distance decreased from 1.10 to 1.05; for NeuS, it improved from 1.16 to 1.05.

Qualitative Improvements

Visually, BundleRecon excels at capturing fine textures and maintaining geometric integrity. It successfully fills "holes" in meshes where traditional point-based methods fail and resolves intricate details like concave stone surfaces or figurine hair.

Qualitative comparison showing how BundleRecon (right) fills geometric holes present in baseline reconstructions (left).

Conclusion and Future Work

BundleRecon demonstrates that considering local pixel neighborhoods significantly enhances the fidelity of neural implicit reconstructions. While the method introduces a higher memory cost due to processing $s^2$ rays per bundle, this can be managed by reducing the total number of bundles. Future research will focus on further memory optimization and utilizing ray bundles to directly constrain geometry normals for even higher precision.

Summary updated

BibTeX
@misc{zhang2023bundlerecon,
  author = {Weikun Zhang and Jianke Zhu},
  title = {{BundleRecon: Ray Bundle-Based 3D Neural Reconstruction}},
  year = {2023},
  eprint = {2305.07342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.07342},
}
Figure