TL;DR
This paper introduces DetRF, a novel method for synthesizing novel views of dynamic scenes from casual monocular videos. DetRF effectively disentangles static backgrounds from dynamic objects by employing a "backdrop-driven" neural radiance field and an occlusion weight module, leading to cleaner and more accurate renderings.
What Problem Does This Paper Address?
Synthesizing new views of dynamic scenes from limited input, especially from a single camera (monocular video), is a long-standing challenge. Existing methods often struggle with accurately representing both static and moving elements, particularly in occluded areas or when dealing with the underlying scene distribution. Many approaches rely on complex multi-camera setups or struggle to generalize beyond specific domains. The core problem is how to generate photorealistic novel views of dynamic scenes that maintain both detailed textures and accurate motion, while also providing a clean separation of static background elements.
Methods
DetRF tackles this by proposing a two-pronged neural representation:
- Backdrop Representation: This component learns an implicit distribution of the static background. It uses a "backdrop encoder" to capture this distribution and a latent code that drives the rendering of static elements. An attention mechanism is employed to exploit this latent variable for better scene understanding.
- Dynamic NeRF: A separate neural radiance field is used to represent the time-varying dynamic objects. This component takes a 6D input (3D location and viewing direction) and also incorporates temporal information.
- Occlusion Weight Module: This is a key innovation. It generates an "occlusion weight" for each ray sample. This weight helps to determine the transmittance contribution from either the static background or the dynamic objects. This module is crucial for decoupling the two components and handling occlusions more effectively. The final rendering is a blend of the static and dynamic renderings, weighted by this occlusion information.
- Loss Functions: To optimize this complex system, DetRF uses several regularization losses, including reconstruction loss, transmittance loss (to encourage clear separation), and depth/flow consistency losses to leverage geometric priors.
Datasets
- NVIDIA dynamic scenes: A dataset comprising 4 scenes captured by a hand-held monocular camera and 8 scenes from a multi-view setup.
- Urban driving scenes (Argoverse): A newly built dataset of real-world urban street scenes captured from an autonomous driving vehicle's monocular cameras. These scenes feature dynamic actions of vehicles and people.
Key Insights and Ideas
- Decoupling static and dynamic is key: The paper highlights that effectively separating static backgrounds from dynamic objects is crucial for high-quality novel view synthesis in dynamic scenes.
- Occlusion weight for transmittance: Introducing a ray-level occlusion weight that influences transmittance is a novel way to manage the blending of static and dynamic scene components, leading to better disentanglement.
- Backdrop encoder for static scenes: Leveraging a dedicated encoder for the static background allows for a more robust representation of persistent scene elements, which are often overlooked by methods focusing solely on temporal dynamics.
- Monocular video is sufficient: The success of DetRF on monocular video inputs demonstrates the potential for high-quality dynamic scene synthesis without requiring expensive multi-camera setups.
Key Results
DetRF achieves state-of-the-art performance on both NVIDIA dynamic scenes and their newly introduced urban driving dataset.
- Quantitative: DetRF consistently outperforms baseline methods (including Yoon et al., NeRF, NSFF, and Gao et al.) in terms of PSNR and LPIPS metrics. For instance, on urban driving scenes, DetRF achieves a PSNR of 33.16 and an LPIPS of 0.0359, outperforming the next best method by a noticeable margin. Ablation studies show that each component (backdrop encoder, occlusion weight module, depth/flow losses) contributes significantly to the overall performance.
- Qualitative: The method produces visually superior results, with more accurate texture recovery in both static and motion regions. It also demonstrates a remarkable ability to disentangle clean static backgrounds from dynamic scenes, as shown in Figure 6, producing uncontaminated RGB and depth outputs.
Limitations and Future Work
- Limitations:
- The method's performance is sensitive to the accuracy of input camera poses and optical flow estimations. Inaccurate priors can lead to artifacts like ghosting in motion areas or incomplete disentanglement (Figure 9).
- Like many NeRF-based methods, it can be computationally intensive.
- Future Work:
- The authors suggest exploring applications in autonomous driving scene simulation and editor models for outdoor dynamic scenes.
- Further research into improving the robustness to inaccurate input priors is also a promising direction.
Summary updated