Dynamic Neural Radiance Field From Defocused Monocular Video

Xianrui Luo, Huiqiang Sun, Juewen Peng, Zhiguo Cao

Introduction

Neural Radiance Fields (NeRFs) have fundamentally changed the landscape of 3D scene reconstruction and novel view synthesis. While recent advancements have extended NeRFs to dynamic environments, most existing frameworks rely on a "perfect camera" assumption, where every pixel in the input video is assumed to be in sharp focus.

In real-world scenarios—particularly when using smartphones or professional cameras with wide apertures—defocus blur is an inherent physical phenomenon. When subjects move outside the camera's focal plane, they become blurred, creating a "bokeh" effect. For standard dynamic NeRFs, this blur is catastrophic; the lack of high-frequency detail makes it nearly impossible to maintain temporal consistency or reconstruct fine textures. While deblurring NeRFs exist for static scenes, $D^2RF$ is the first framework specifically designed to reconstruct sharp, all-in-focus 3D scenes from defocused monocular dynamic videos.

Methodology

The $D^2RF$ framework integrates a physical defocus model directly into the NeRF optimization process through three primary technical innovations:

1. Layered DoF Volume Rendering

The authors identified a mathematical bridge between volume rendering and classical Depth-of-Field (DoF) principles. In traditional rendering, DoF is often achieved by blending layers based on their visibility. $D^2RF$ recognizes that the opacity used in NeRF volume rendering is physically analogous to this layer visibility. By fusing these concepts, the pipeline can "predict" blur during the training phase. This allows the underlying 3D representation to remain sharp while the rendered output is optimized to match the blurry input frames.

2. Ray-based Sparse Kernels

Standard image deblurring typically employs 2D kernels (such as disk or Gaussian blur). However, because NeRF operates on individual rays, applying dense 2D kernels is computationally prohibitive. To solve this, the authors transitioned to an optimized sparse ray kernel. They utilize a small Multi-Layer Perceptron (MLP) to predict the 3D positions and weights of a minimal set of rays—typically just five—to simulate the blur. This approach drastically reduces the "ray explosion" that would otherwise occur, making the deblurring process computationally feasible.

3. Cross-time Rendering and Blending

To handle dynamic motion, the model employs a dual-MLP architecture: one for the static background and another for moving objects. The system uses scene flow to warp rays across different timestamps. This ensures that the reconstructed scene remains spatially and temporally consistent, even if the camera's focus shifts or the subject moves significantly during the video sequence.

Experimental Evaluation

Datasets

Due to the lack of existing benchmarks for defocused dynamic scenes, the researchers curated a new dataset:

  • Synthesized VDW Dataset: Derived from the Video Depth World (VDW) stereo dataset, using the "BokehMe" pipeline to generate realistic defocus blur based on ground-truth disparity maps across eight diverse scenes.
  • Real-world Video: The model was also tested on handheld, real-world defocused footage to validate its robustness in non-simulated environments.

Results

$D^2RF$ demonstrated quantitative superiority over state-of-the-art dynamic NeRFs such as NSFF and HyperNeRF. On the synthesized dataset, $D^2RF$ achieved a PSNR of 27.30, outperforming the next best baseline (~27.01). Qualitatively, the model successfully recovers sharp textures—such as text on signs or intricate mechanical details—that are typically smeared by other methods. Furthermore, while 2D deblurring baselines often suffer from temporal "flickering," $D^2RF$ produces smooth, consistent video when moving a virtual camera through the reconstructed 3D space.

Insights and Limitations

A key insight of this work is that 3D-aware deblurring is inherently superior to 2D pre-processing. Naive 2D deblurring tools are often "view-inconsistent," meaning the deblurred pixels in one frame do not align with those in another when projected into 3D space. By performing deblurring inside the 3D optimization loop, $D^2RF$ ensures physical plausibility.

However, the method does face certain limitations:

  • Extreme Blur: If the input is so defocused that information is almost entirely lost, the model cannot "hallucinate" missing details.
  • Computational Requirements: The multi-ray sampling increases memory overhead, requiring approximately 20GB of VRAM compared to the 12GB typically needed for standard dynamic NeRFs.

Conclusion

$D^2RF$ represents a significant step forward in processing real-world video for 3D reconstruction. By linking the physics of depth-of-field with volume rendering and utilizing efficient sparse kernels, it enables the creation of high-quality, all-in-focus content from imperfect, real-world monocular sources. Future work may explore explicit representations, such as 3D Gaussian Splatting, to further improve training speed and memory efficiency.

Summary updated

BibTeX
@inproceedings{luo2024dynamic,
  author = {Xianrui Luo and Huiqiang Sun and Juewen Peng and Zhiguo Cao},
  title = {{Dynamic Neural Radiance Field From Defocused Monocular Video}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72652-1\_9},
}
Figure