Learning with Unreliability: Fast Few-shot Voxel Radiance Fields with Relative Geometric Consistency

Yingjie Xu, Bangzhen Liu, Hao Tang, Bailin Deng, Shengfeng He

Introduction

Neural Radiance Fields (NeRF) have revolutionized the field of 3D scene reconstruction, enabling the synthesis of photorealistic novel views from a set of 2D images. However, NeRF’s success is typically contingent on having a dense set of input views—often dozens or hundreds. When restricted to a "few-shot" scenario (e.g., 3 to 9 images), standard NeRF models struggle with severe overfitting, manifesting as "floaters" in empty space and broken, inconsistent geometry.

To mitigate this, researchers frequently employ pseudo-view synthesis, a technique where existing images are warped to unobserved viewpoints to provide additional supervision. While effective in theory, warping is imperfect; it creates "black holes" in occluded areas and introduces noisy artifacts. Most current methods simply mask these unreliable regions out, discarding potentially valuable geometric information. This paper introduces ReVoRF (Relative Geometric Consistency Voxel Radiance Fields), a framework that argues that even "unreliable" data contains useful signals if interpreted through the lens of relative geometry.

The ReVoRF Framework

ReVoRF is a voxel-based optimization framework designed for speed and robustness. Unlike traditional methods that treat data reliability as a binary (valid or invalid), ReVoRF treats it as a spectrum. The core of the approach lies in how it processes synthesized views and regularizes the underlying 3D representation.

Bilateral Geometric Consistency

The primary innovation is the Bilateral Geometric Consistency Loss. During training, the model categorizes synthesized views into reliable and unreliable regions:

  • Reliable Regions: These areas are supervised using standard color and density losses, as the warped data is considered high-fidelity.
  • Unreliable Regions: In areas where warping fails (such as disoccluded holes), the model applies a relative depth consistency term. Rather than attempting to predict exact pixel colors for these gaps, the model ensures that the relative depth relationships between neighboring pixels match a prior generated by a pretrained depth estimation model (DPT). This allows the model to "inpain the geometry" even when visual color data is missing.

Reliability-Aware Voxel Smoothing

While voxel grids (like those used in DVGO) offer significant speed advantages over pure MLP-based NeRFs, they are highly susceptible to overfitting in few-shot settings. ReVoRF addresses this by calculating a "reliability score" for every voxel based on the density of reliable rays passing through it. Voxels in low-certainty areas are subjected to more aggressive smoothing, preventing the formation of artifacts and maintaining structural integrity across the scene.

Reliability-Guided Learning

The training curriculum is designed to be adaptive. The model initially prioritizes learning from high-confidence, reliable zones. As the optimization progresses, it gradually incorporates information from the unreliable regions, ensuring that the initial geometric backbone is stable before attempting to resolve complex, occluded areas.

Experimental Results

The authors evaluated ReVoRF on two benchmark datasets: the Realistic Synthetic 360° dataset (8 scenes with complex, non-Lambertian materials) and the LLFF (Local Light Field Fusion) dataset (8 real-world, forward-facing scenes).

Performance and Accuracy

ReVoRF demonstrates a significant improvement in both reconstruction quality and computational efficiency:

  • State-of-the-Art Accuracy: The model achieved a 5% improvement in PSNR (Peak Signal-to-Noise Ratio) over previous leading few-shot methods like GeCoNeRF. On the Synthetic 360° benchmark, it reached a PSNR of 20.72, compared to GeCoNeRF's 19.78.
  • Rapid Training: By leveraging voxel-based optimization, ReVoRF can reconstruct a 360° scene in as little as 7 minutes. This is orders of magnitude faster than many competing few-shot approaches that require hours or days of training.
  • Rendering Speed: The optimized voxel representation allows for real-time rendering at approximately 3 FPS, making it practical for interactive applications.

Discussion and Limitations

The fundamental insight of ReVoRF is that relative depth is more robust than absolute color. By leveraging the relative consistency of depth priors in disoccluded areas, the model successfully fills geometric gaps that other models leave as "cloudy" artifacts.

However, the reliance on voxel grids and aggressive smoothing does present trade-offs. The model can sometimes produce slightly over-smoothed results, leading to the loss of fine, high-frequency textural details. Additionally, the memory requirements of dense voxel grids make it challenging to scale the current approach to extremely large or unbounded outdoor environments.

Conclusion

ReVoRF represents a significant step forward in few-shot 3D reconstruction. By shifting the focus from discarding unreliable data to extracting relative geometric truths, it achieves a superior balance of speed and visual fidelity. Future work may involve hybrid representations—combining voxels with point clouds or implicit surfaces—to better preserve high-frequency details while maintaining the rapid convergence times demonstrated here.

Summary updated

BibTeX
@inproceedings{xu2024learning,
  author = {Yingjie Xu and Bangzhen Liu and Hao Tang and Bailin Deng and Shengfeng He},
  title = {{Learning with Unreliability: Fast Few-shot Voxel Radiance Fields with Relative Geometric Consistency}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01923},
}
Figure