Abstract
We present Non-Rigid Neural Radiance Fields (NR-NeRF), a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. Our approach takes RGB images of a dynamic scene as input (e.g., from a monocular video recording), and creates a high-quality space-time geometry and appearance representation. We show that a single handheld consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views, e.g. a ‘bullet-time’ video effect. NR-NeRF disentangles the dynamic scene into a canonical volume and its deformation. Scene deformation is implemented as ray bending, where straight rays are deformed non-rigidly. We also propose a novel rigidity network to better constrain rigid regions of the scene, leading to more stable results. The ray bending and rigidity network are trained without explicit supervision. Our formulation enables dense correspondence estimation across views and time, and compelling video editing applications such as motion exaggeration. Our code will be open sourced.
Introduction
Neural Radiance Fields (NeRF) have fundamentally changed the landscape of 3D reconstruction by using coordinate-based neural networks to represent complex scenes. However, the original NeRF formulation assumes a perfectly static environment. When applied to dynamic scenes—where objects or people move during recording—the model typically fails, resulting in severe "ghosting" artifacts or blur.
Capturing dynamic scenes traditionally requires expensive multi-view camera rigs to capture the same moment from many angles simultaneously. Reconstructing these scenes from a single, moving camera (the "monocular" setting) is a severely under-constrained problem because both the scene's geometry and its appearance change at every frame. NR-NeRF addresses this challenge, enabling high-quality reconstruction of general non-rigid scenes without requiring specialized templates (like pre-defined human models) or complex multi-camera setups.
Methodology
The core innovation of NR-NeRF is the disentanglement of the dynamic scene into two distinct components: a static canonical volume and a time-varying deformation field.
Canonical Volume and Ray Bending
Instead of warping the 3D geometry directly, which can be computationally intensive and unstable, the model implements "ray bending." A static NeRF (Multi-Layer Perceptron) represents the geometry and appearance of the scene in a "base" or canonical state. When the camera shoots a straight ray into the scene at a specific time, a separate ray-bending MLP calculates how that ray should curve to map points from the deformed state back into the canonical volume. This allows the system to maintain the inherent consistency of the NeRF framework while accommodating fluid motion.
Unsupervised Constraints
To make this work without direct supervision, the authors introduced several key physical priors:
- Rigidity Network: A small MLP predicts a "rigidity score" for every point in space. This allows the model to automatically segment the scene into a rigid background (which should remain static) and a non-rigid foreground. This segmentation is learned entirely without labels, as the model "realizes" that static parts of the scene are easier to represent without deformation.
- Divergence Loss: Inspired by fluid dynamics, this loss encourages the deformation field to be volume-preserving. It prevents the model from "cheating" by shrinking or expanding geometry to fit pixels, which is crucial for rendering parts of the scene that were partially occluded during the video.
- Offset Regularization: This loss function encourages sparse and small deformations, ensuring the model remains stable and does not overfit to transient noise.
Key Results and Applications
NR-NeRF demonstrates superior stability compared to previous baselines like Neural Volumes. While naive dynamic NeRF implementations often suffer from severe temporal flickering, NR-NeRF produces significantly sharper foregrounds and more stable backgrounds. Quantitatively, the method achieves state-of-the-art scores in perceptual metrics such as SSIM and LPIPS across a variety of real-world handheld captures.
Beyond simple reconstruction, the disentangled representation enables several creative video editing applications:
- Foreground Removal: By disabling the non-rigid components, users can effectively "see through" moving subjects to the reconstructed background.
- Motion Exaggeration: By scaling the deformation offsets calculated by the ray-bending network, movements can be made more dramatic or dampened.
- Time Interpolation: Because the deformation is conditioned on a continuous latent code, the model can generate smooth slow-motion effects by interpolating between captured frames.
Discussion and Limitations
While NR-NeRF represents a significant step forward, it shares some limitations with the original NeRF, notably slow training and rendering times. The model also assumes Lambertian materials and constant lighting, which can lead to artifacts on shiny surfaces or in scenes with moving shadows. Furthermore, the method relies on a dominant static background to calculate camera poses via Structure-from-Motion (SfM); if the camera is static and only the subject moves, the system cannot currently reconstruct the 3D geometry. Future work will likely focus on improving rendering speed and handling more complex lighting environments.
Summary updated