Introduction
Neural Radiance Fields (NeRF) have fundamentally changed how we approach 3D reconstruction and novel view synthesis. However, standard NeRF models suffer from "viewpoint hunger," requiring a dense set of camera angles to produce accurate geometry. In sparse-data scenarios—common in robotics or casual consumer photography—these models often fall victim to the shape-radiance ambiguity. To satisfy the color requirements of training images without sufficient geometric constraints, the model creates "floaters" or cloudy geometry near the camera, leading to degraded depth maps and artifacts in novel views.
DeLiRa (Depth, Light, and Radiance) addresses this challenge by bridging the gap between volumetric rendering and self-supervised depth estimation. By integrating multi-view photometric consistency into a unified framework, DeLiRa achieves state-of-the-art geometry and rendering quality even with limited viewpoints.
Methodology
DeLiRa utilizes a unified architecture that decodes three distinct scene representations from a shared set of latent codes using a Transformer-based cross-attention decoder:
- Radiance Field: A volumetric representation of color and density for high-fidelity rendering.
- Light Field: A fast representation mapping oriented rays directly to colors, bypassing expensive volumetric sampling.
- Depth Field: A representation mapping rays directly to depth values.
Multi-view Photometric Regularization
The core innovation of DeLiRa is its use of predicted depth to enforce physical plausibility. Unlike standard NeRF, which only compares predicted pixels to ground truth in the same image, DeLiRa uses the predicted depth to warp neighboring "context" images into the target view. By minimizing the difference between these warped images and the target, the model is forced to learn geometry that is consistent across multiple viewpoints.
Synergistic Training and Guidance
The framework employs several strategies to optimize performance and speed:
- Virtual Supervision: The volumetric radiance field acts as a teacher, providing pseudo-labels for the faster Light and Depth fields.
- Depth Field Guidance (DFG): Once the Depth Field matures, it guides the volumetric renderer to sample points only where surfaces likely exist, drastically reducing network calls.
- Annealing Schedule: To account for non-Lambertian surfaces (reflections), the photometric loss is gradually phased out, allowing the model to first lock in geometry and then refine complex view-dependent effects.
Experimental Results
The model was evaluated on the ScanNet benchmark, focusing on challenging subsets like ScanNet-Frontal (limited diversity) and ScanNet-Rooms (large scale). DeLiRa outperformed heavyweights like DS-NeRF and NerfingMVS without requiring pre-trained depth networks or sparse point clouds.
Key Performance Metrics
- Depth Accuracy: On ScanNet-Frontal, DeLiRa improved depth accuracy (AbsRel) by 86.3% over standard NeRF and 14.2% over previous state-of-the-art methods.
- Image Quality: The model achieved an 11.1% improvement in PSNR for novel view synthesis compared to methods using pre-trained depth priors.
- Rendering Speed: Through Depth Field Guidance, volumetric rendering speed increased from 0.82 FPS to 37.3 FPS. The dedicated Light Field head achieved real-time performance at over 118 FPS.
Conclusion
DeLiRa demonstrates that explicit geometric consistency is a powerful, "free" supervisor for volumetric rendering. By sharing a latent space across radiance, light, and depth tasks, the model breaks the shape-radiance ambiguity and enables high-quality, real-time 3D reconstruction from sparse data. Future work may extend this approach to large-scale urban environments or generalizable architectures that work across diverse scenes without per-scene retraining.
Summary updated