Depth Priors in Removal Neural Radiance Fields

Zhihao Guo, Peng WangLecture Notes in Computer Science 2024 · DOI · OpenAlex

Abstract

Neural Radiance Fields (NeRF) have transformed 3D reconstruction, yet efficient object removal remains a significant challenge due to the reliance on computationally expensive depth priors. This paper presents a streamlined pipeline that integrates monocular depth estimation—specifically ZoeDepth—into the SpinNeRF framework. By replacing traditional Structure-from-Motion (SfM) depth generation with deep-learning-based monocular priors, the proposed method achieves a 75x speedup in depth prior generation and a 6.87% improvement in PSNR. These results demonstrate that dense monocular priors not only accelerate the editing workflow but also enhance the visual fidelity of inpainted 3D scenes, making high-fidelity digital twin editing more viable.

1. Introduction

Neural Radiance Fields (NeRF) have revolutionized the synthesis of novel views from sparse 2D images. However, the task of "editing" these fields—specifically removing objects and cleanly inpainting the resulting voids—remains a significant hurdle. To maintain consistency across multiple viewpoints, object removal requires accurate depth priors to guide the inpainting process. Traditionally, these priors are derived from expensive LiDAR hardware or time-consuming Structure-from-Motion (SfM) software like COLMAP. This paper explores an alternative: leveraging modern monocular depth estimation to provide fast, dense, and accurate depth priors for NeRF-based object removal.

2. The Challenge of Depth Priors in 3D Editing

When an object is removed from a 3D scene, the model must hallucinate the background geometry and texture that was previously occluded. Without a reliable depth map, the inpainting algorithm cannot ensure that the "filled-in" area remains spatially consistent as the camera moves.

Existing solutions face a trade-off between accessibility and efficiency:

  1. LiDAR: Provides high-accuracy ground truth but is cost-prohibitive and requires specialized hardware.
  2. COLMAP (SfM): A common software-based alternative that is free but suffers from extreme latency and often produces "sparse" data, leaving gaps in the depth map that can lead to artifacts in the final NeRF reconstruction.

3. Methodology

The proposed pipeline builds upon the SpinNeRF framework, a state-of-the-art model for 3D scene editing. The core innovation is the replacement of traditional depth-sensing methods with ZoeDepth, a deep learning model capable of predicting dense depth maps from single images.

3.1 Monocular Depth Integration

Instead of relying on multi-view geometry to calculate depth, the system utilizes ZoeDepth to generate a dense depth prior for each input frame. The researchers compared several monocular models, including EcoDepth and DepthAnything, finding that ZoeDepth provided the best balance of sharp edge definition and metric consistency required for 3D stability.

3.2 Validation Framework

To ensure that monocular depth could reliably replace traditional methods, the authors performed a benchmark study using the KITTI dataset. They first validated that COLMAP’s dense reconstruction is accurate enough to serve as a ground truth substitute when LiDAR is unavailable, achieving a $\delta_1$ accuracy of 91%. This established a rigorous baseline for evaluating the monocular priors.

3.3 Inpainting and Optimization

The pipeline takes sparse views and user-defined masks as input. ZoeDepth generates the depth priors, which are then utilized by a multi-view consistent inpainting process (using the LAMA inpainter). The NeRF is optimized using a combination of Perceptual Loss (LPIPS) and a specific Depth Loss ($\mathcal{L}_{depth}$) to ensure the rendered geometry aligns with the predicted priors.

4. Experimental Results

The performance of the ZoeDepth-augmented pipeline was evaluated on the SpinNeRF dataset, which includes 10 diverse real-world scenes.

4.1 Efficiency Gains

The most significant impact was observed in processing speed. Traditional depth prior generation (DSNeRF/COLMAP) required approximately 44.5 seconds per image. By contrast, the ZoeDepth integration reduced this to just 0.58 seconds per image, representing a 75x speedup.

4.2 Quality Improvements

Beyond speed, the use of dense monocular priors improved the visual quality of the reconstruction. The pipeline achieved a 6.87% improvement in PSNR (increasing from 21.943 to 23.451) compared to the original SpinNeRF implementation. This improvement is attributed to the "cleaner" signal provided by monocular models; unlike the noisy or incomplete maps generated by sparse SfM, ZoeDepth provides a smooth, continuous depth field that allows the NeRF to learn background geometry more effectively.

5. Conclusion

By integrating monocular depth estimation into the NeRF editing workflow, this research demonstrates that it is possible to achieve both higher efficiency and superior visual fidelity. The transition from sparse SfM to dense monocular priors removes a significant bottleneck in the 3D editing pipeline. While NeRF training times remain a factor for real-time applications, this approach moves the field closer to instantaneous "click-and-remove" interfaces for digital twins and virtual environments. Future work may explore the integration of these priors with faster rendering techniques, such as Gaussian Splatting, to further reduce the computational overhead.

Summary updated

BibTeX
@inproceedings{guo2024depth,
  author = {Zhihao Guo and Peng Wang},
  title = {{Depth Priors in Removal Neural Radiance Fields}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2024},
  doi = {10.1007/978-3-031-72059-8\_31},
}

Loading related content...

Figure