Introduction
Neural implicit representations, such as NeRF, NeuS, and IDR, have revolutionized 3D reconstruction from 2D images. Despite their success, these methods typically rely on sampling individual, discrete rays. This approach captures longitudinal data along the ray but ignores transversal information—the relationship between a pixel and its neighbors. This "tunnel vision" often results in noisy geometry, missing details, or physical "holes" in the reconstructed mesh, particularly when pixel sampling is sparse.
Methodology
The authors introduce BundleRecon, a module designed to integrate into existing neural reconstruction pipelines. The core shift is from point-based sampling to patch-based sampling.
Ray Bundle Construction
Instead of sampling $n$ individual rays, BundleRecon samples $n$ patches of $s \times s$ pixels (e.g., $3 \times 3$ bundles). This is analogous to using a "flashlight" to illuminate a surface area rather than a "laser pen" to pinpoint a single dot.
Bundle-Based Loss Functions
To leverage the spatial information within these patches, three new loss components are introduced:
- Mean and Variance Loss ($L_m, L_v$): These ensure the statistical color distribution of the rendered patch aligns with the ground truth, acting as a regularizer for smoother surfaces.
- Convolutional Loss ($L_{conv}$): By applying a Sobel kernel to the patches, the model extracts edge and texture features, providing structural signals that single-pixel methods lack.
- Distance Masking ($M_P$): A spatial distance mask filters out irrelevant background information when a bundle overlaps an object boundary, preventing geometric confusion.
Experiments and Results
BundleRecon was evaluated on the DTU and BlendedMVS datasets. It demonstrated broad compatibility, functioning as a plug-in for both IDR (ray tracing) and NeuS (volume rendering).
Quantitative Performance
On the DTU dataset, BundleRecon improved the mean Chamfer distance—a key metric for 3D accuracy. When added to IDR, the distance decreased from 1.10 to 1.05; for NeuS, it improved from 1.16 to 1.05.
Qualitative Improvements
Visually, BundleRecon excels at capturing fine textures and maintaining geometric integrity. It successfully fills "holes" in meshes where traditional point-based methods fail and resolves intricate details like concave stone surfaces or figurine hair.
Conclusion and Future Work
BundleRecon demonstrates that considering local pixel neighborhoods significantly enhances the fidelity of neural implicit reconstructions. While the method introduces a higher memory cost due to processing $s^2$ rays per bundle, this can be managed by reducing the total number of bundles. Future research will focus on further memory optimization and utilizing ray bundles to directly constrain geometry normals for even higher precision.
Summary updated