DeLiRa: Self-Supervised Depth, Light, and Radiance Fields

Vitor Guizilini, Igor Vasiljevic, Jiading Fang, Rares Ambrus, Sergey Zakharov, Vincent Sitzmann, Adrien Gaidon

Introduction

Neural Radiance Fields (NeRF) have fundamentally changed how we approach 3D reconstruction and novel view synthesis. However, standard NeRF models suffer from "viewpoint hunger," requiring a dense set of camera angles to produce accurate geometry. In sparse-data scenarios—common in robotics or casual consumer photography—these models often fall victim to the shape-radiance ambiguity. To satisfy the color requirements of training images without sufficient geometric constraints, the model creates "floaters" or cloudy geometry near the camera, leading to degraded depth maps and artifacts in novel views.

DeLiRa (Depth, Light, and Radiance) addresses this challenge by bridging the gap between volumetric rendering and self-supervised depth estimation. By integrating multi-view photometric consistency into a unified framework, DeLiRa achieves state-of-the-art geometry and rendering quality even with limited viewpoints.

Methodology

DeLiRa utilizes a unified architecture that decodes three distinct scene representations from a shared set of latent codes using a Transformer-based cross-attention decoder:

  1. Radiance Field: A volumetric representation of color and density for high-fidelity rendering.
  2. Light Field: A fast representation mapping oriented rays directly to colors, bypassing expensive volumetric sampling.
  3. Depth Field: A representation mapping rays directly to depth values.

Multi-view Photometric Regularization

The core innovation of DeLiRa is its use of predicted depth to enforce physical plausibility. Unlike standard NeRF, which only compares predicted pixels to ground truth in the same image, DeLiRa uses the predicted depth to warp neighboring "context" images into the target view. By minimizing the difference between these warped images and the target, the model is forced to learn geometry that is consistent across multiple viewpoints.

The DeLiRa framework combines view synthesis and multi-view photometric losses to regularize the shared latent space.

Synergistic Training and Guidance

The framework employs several strategies to optimize performance and speed:

  • Virtual Supervision: The volumetric radiance field acts as a teacher, providing pseudo-labels for the faster Light and Depth fields.
  • Depth Field Guidance (DFG): Once the Depth Field matures, it guides the volumetric renderer to sample points only where surfaces likely exist, drastically reducing network calls.
  • Annealing Schedule: To account for non-Lambertian surfaces (reflections), the photometric loss is gradually phased out, allowing the model to first lock in geometry and then refine complex view-dependent effects.

Experimental Results

The model was evaluated on the ScanNet benchmark, focusing on challenging subsets like ScanNet-Frontal (limited diversity) and ScanNet-Rooms (large scale). DeLiRa outperformed heavyweights like DS-NeRF and NerfingMVS without requiring pre-trained depth networks or sparse point clouds.

A sample depth map generated by DeLiRa, demonstrating the model's ability to recover clean, sharp geometry from sparse views.

Key Performance Metrics

  • Depth Accuracy: On ScanNet-Frontal, DeLiRa improved depth accuracy (AbsRel) by 86.3% over standard NeRF and 14.2% over previous state-of-the-art methods.
  • Image Quality: The model achieved an 11.1% improvement in PSNR for novel view synthesis compared to methods using pre-trained depth priors.
  • Rendering Speed: Through Depth Field Guidance, volumetric rendering speed increased from 0.82 FPS to 37.3 FPS. The dedicated Light Field head achieved real-time performance at over 118 FPS.

Conclusion

DeLiRa demonstrates that explicit geometric consistency is a powerful, "free" supervisor for volumetric rendering. By sharing a latent space across radiance, light, and depth tasks, the model breaks the shape-radiance ambiguity and enables high-quality, real-time 3D reconstruction from sparse data. Future work may extend this approach to large-scale urban environments or generalizable architectures that work across diverse scenes without per-scene retraining.

Summary updated

Abstract

Differentiable volumetric rendering is a powerful paradigm for 3D reconstruction and novel view synthesis. However, standard volume rendering approaches struggle with degenerate geometries in the case of limited viewpoint diversity, a common scenario in robotics applications. In this work, we propose to use the multi-view photometric objective from the self-supervised depth estimation literature as a geometric regularizer for volumetric rendering, significantly improving novel view synthesis without requiring additional information. Building upon this insight, we explore the explicit modeling of scene geometry using a generalist Transformer, jointly learning a radiance field as well as depth and light fields with a set of shared latent codes. We demonstrate that sharing geometric information across tasks is mutually beneficial, leading to improvements over single-task learning without an increase in network complexity. Our DeLiRa architecture achieves state-of-the-art results on the ScanNet benchmark, enabling high quality volumetric rendering as well as real-time novel view and depth synthesis in the limited viewpoint diversity setting.
BibTeX
@inproceedings{guizilini2023delira,
  author = {Vitor Guizilini and Igor Vasiljevic and Jiading Fang and Rares Ambrus and Sergey Zakharov and Vincent Sitzmann and Adrien Gaidon},
  title = {{DeLiRa: Self-Supervised Depth, Light, and Radiance Fields}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure