Detachable Novel Views Synthesis of Dynamic Scenes Using Distribution-Driven Neural Radiance Fields

Boyu Zhang, Wenbo Xu, Zheng Zhu, Guan HuangarXiv 2023 · OpenReview

TL;DR

This paper introduces D⁴NeRF, a novel method for generating realistic novel views of dynamic scenes from single, casual monocular videos. It achieves this by decoupling static backgrounds from dynamic elements using a distribution-driven approach and an occlusion-weighting mechanism, leading to cleaner renderings and better detail recovery.

What Problem Does This Paper Address?

Synthesizing new viewpoints of dynamic scenes from limited video input is a notoriously difficult problem. Existing methods often struggle with accurately representing static backgrounds and handling occlusions between moving objects. They tend to either rely on complex multi-camera setups, miss crucial background distribution information, or fail to properly account for how light is blocked (transmittance) along a ray, leading to artifacts like ghosting or blurry details, especially in static areas. This limits their applicability in real-world scenarios like autonomous driving or augmented reality.

Methods

D⁴NeRF tackles this by introducing a two-pronged neural representation:

  1. Distribution-Driven Background Representation: This component learns an implicit distribution of the static background. The idea is that static elements, even when viewed from different angles, share underlying structural properties. This representation is designed to be robust to viewpoint changes.
  2. 6D-Input NeRF for Dynamic Objects: A standard NeRF-like network is used to capture the time-varying aspects of the scene, representing dynamic objects.
  3. Occlusion Weight Module: This is a key innovation. For each ray cast through the scene, an "occlusion weight" is generated. This weight indicates the transmittance (how much light passes through) for both static and dynamic components along that ray. This allows the method to explicitly model and disentangle contributions from static and dynamic elements, leading to cleaner separation and better handling of occlusions.

The final rendering is achieved by blending the outputs of these two components, guided by the occlusion weights. The training process involves several loss functions, including reconstruction loss, distribution loss, and regularization losses for depth and optical flow to ensure temporal consistency and accurate geometry.

Datasets

The authors evaluate D⁴NeRF on two types of datasets:

  • NVIDIA Dynamic Scenes: A public dataset comprising both hand-held monocular and synchronized multi-view camera recordings.
  • Urban Driving Scenes: A new dataset built from the Argoverse dataset, specifically curated for dynamic novel view synthesis. This dataset features urban street scenes captured by an autonomous driving vehicle with multiple cameras.

Key Insights and Ideas

  • Decoupling Static and Dynamic is Key: The core insight is that separating the static background from dynamic elements is crucial for high-quality novel view synthesis in complex scenes.
  • Background Distribution Matters: Simply modeling dynamic elements isn't enough. Understanding the underlying distribution of static elements provides a more robust foundation for rendering.
  • Transmittance as an Occlusion Cue: Explicitly modeling ray transmittance, rather than just relying on RGB color blending, provides a more principled way to handle occlusions and disentangle scene components.
  • Attention for Latent Variable Exploitation: The use of attention mechanisms helps effectively leverage the sampled latent variables from the background distribution.

Key Results

  • Superior Novel View Synthesis: D⁴NeRF consistently outperforms existing methods (including NeRF, NSFF, and Gao et al.) in terms of PSNR, SSIM, and LPIPS metrics on both NVIDIA dynamic scenes and their custom urban driving dataset.
  • Cleaner Static Backgrounds: The method excels at producing uncontaminated static backgrounds, even in challenging scenarios with occluded regions.
  • Improved Texture and Motion Detail: D⁴NeRF demonstrates better recovery of fine textures and motion details compared to baselines. For instance, it significantly reduces ghosting artifacts seen in other methods.
  • Effective Module Contributions: Ablation studies show that each proposed module (distribution encoder, occlusion weight, depth/flow losses) contributes significantly to the overall performance.

Limitations and Future Work

  • Reliance on Accurate Input: Like many NeRF-based methods, D⁴NeRF's performance is sensitive to the accuracy of input camera poses and optical flow estimations. Inaccurate inputs can lead to motion artifacts and incomplete decoupling.
  • Future Directions: The authors suggest exploring more advanced decomposition techniques and editor models for 3D dynamic scenes. They also mention the potential of using deformable latent codes for more sophisticated dynamic object representations, similar to other recent works.

Summary updated

Abstract

Representing and synthesizing novel views in real-world dynamic scenes from casual monocular videos is a long-standing problem. Existing solutions typically approach dynamic scenes by applying geometry techniques or utilizing temporal information between several adjacent frames without considering the underlying background distribution in the entire scene or the transmittance over the ray dimension, limiting their performance on static and occlusion areas. Our approach $\textbf{D}$istribution-$\textbf{D}$riven neural radiance fields offers high-quality view synthesis and a 3D solution to $\textbf{D}$etach the background from the entire $\textbf{D}$ynamic scene, which is called $\text{D}^4$NeRF. Specifically, it employs a neural representation to capture the scene distribution in the static background and a 6D-input NeRF to represent dynamic objects, respectively. Each ray sample is given an additional occlusion weight to indicate the transmittance lying in the static and dynamic components. We evaluate $\text{D}^4$NeRF on public dynamic scenes and our urban driving scenes acquired from an autonomous-driving dataset. Extensive experiments demonstrate that our approach outperforms previous methods in rendering texture details and motion areas while also producing a clean static background. Our code will be released at https://github.com/Luciferbobo/D4NeRF.
BibTeX
@misc{zhang2023detachable,
  author = {Boyu Zhang and Wenbo Xu and Zheng Zhu and Guan Huang},
  title = {{Detachable Novel Views Synthesis of Dynamic Scenes Using Distribution-Driven Neural Radiance Fields}},
  year = {2023},
  eprint = {2301.00411},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.00411},
}
Figure