TL;DR
This paper introduces NR-NeRF, a method that reconstructs and synthesizes novel views of dynamic, non-rigid scenes from a single monocular video. It disentangles scene geometry and appearance from deformation using a novel ray-bending approach and a learned rigidity network, enabling effects like "bullet-time" from casual recordings.
What Problem Does This Paper Address?
Creating realistic free-viewpoint renderings of dynamic scenes has traditionally required complex multi-view capture setups. While recent neural rendering techniques like NeRF excel at static scenes, extending them to dynamic, non-rigid objects (like a person dancing or a cloth waving) from a single, unconstrained camera feed is a significant challenge. This is because each frame captures a different deformed state, violating the assumptions of static scene models. Existing methods often struggle with general objects, lack appearance reconstruction, or require multiple cameras, limiting their applicability to everyday scenarios.
Methods
NR-NeRF tackles this by representing a dynamic scene with two components:
- Canonical Neural Radiance Field: A static representation of the scene's geometry and appearance, learned without direct supervision.
- Scene Deformation Field: This is modeled using a novel "ray bending" technique. Instead of rays traveling in straight lines from the camera, they are deformed non-rigidly. This deformation is learned by an MLP that takes point samples along the ray and a time-specific latent code as input. Crucially, the canonical volume is not conditioned on this deformation, achieving disentanglement.
To improve stability, especially for rigid backgrounds, they introduce a Rigidity Network. This network learns to assign a "rigidity score" to different parts of the scene, allowing deformations to be suppressed in rigid areas. This network is also trained without explicit supervision.
The entire system is trained end-to-end using a reconstruction loss that encourages the rendered novel views to match the input images. Additional regularization losses are employed to encourage smooth deformations and preserve local shape.
Datasets
The paper uses datasets recorded with consumer-grade cameras (Kinect Azure, Blackmagic, phone camera). Specific details about publicly available datasets are not mentioned, but the authors describe their data capture process, noting scenes typically consist of 80-300 images with resolutions of 480x270 or 512x384, captured at 5 fps. They also mention using multi-view data for evaluating the upper quality bound of their approach.
Key Insights and Ideas
- Ray Bending for Deformation: Deforming camera rays rather than the scene volume itself is an effective way to model non-rigid transformations in a neural radiance field context. This allows for a single, static canonical representation.
- Unsupervised Rigidity Learning: A dedicated network to learn scene rigidity without explicit labels is crucial for stabilizing rigid backgrounds and preventing them from deforming along with the foreground. This is a key innovation for handling general scenes with arbitrary backgrounds.
- Disentanglement is Key: Separating the canonical scene representation from the deformation field is essential for achieving high-quality novel view synthesis, especially when the novel camera trajectory significantly differs from the input.
- "Bullet-Time" from Monocular Video: The method demonstrates that sophisticated effects like bullet-time, previously requiring complex setups, are achievable from simple monocular recordings.
Key Results
- High-Quality Novel View Synthesis: NR-NeRF produces sharp and stable novel views of dynamic scenes, outperforming baselines like rigid NeRF and Neural Volumes in terms of SSIM and LPIPS scores.
- Stable Backgrounds: The rigidity network significantly improves background stability in novel views compared to methods without it or with naive regularization.
- Dense Correspondences: The approach implicitly learns dense correspondences between views and time steps, useful for editing.
- Scene Editing Capabilities: The method enables simple editing tasks like foreground removal, motion exaggeration/dampening, and temporal super-sampling.
Limitations and Future Work
- Appearance Changes: The method does not explicitly account for appearance changes due to deformation or lighting variations (e.g., changing shadows).
- Motion Blur: Input motion blur is not modeled and can lead to artifacts.
- Structure-from-Motion Dependency: Accurate camera poses are assumed, which rely on static and dominant backgrounds for structure-from-motion estimation.
- Under-constrained Problem: For highly dynamic scenes and significantly novel views, the problem remains under-constrained, leading to potential trade-offs between sharpness and stability.
- Runtime: Ray bending increases runtime by about 20% compared to standard NeRF.
- Future Work: Extensions to multi-view data and view-dependent effects are suggested for higher quality. Editable neural representations and more robust scene editing are also promising directions.
Summary updated