Abstract: Neural Radiance Fields (NeRF) have set a new standard for 3D scene reconstruction, but they typically require static subjects to avoid artifacts. We present "Nerfies," the first method capable of photorealistically reconstructing non-rigidly deforming scenes using only a casual video from a mobile phone. By introducing a learned deformation field that maps moving subjects into a static canonical template, our approach enables high-fidelity 3D "selfies" without specialized hardware.
Introduction
The advent of Neural Radiance Fields (NeRF) revolutionized 3D reconstruction by representing scenes through neural networks with unprecedented detail. However, NeRF's reliance on a static scene assumption often leads to failure when subjects move, resulting in blur or "ghosting" artifacts. Capturing humans is particularly challenging due to inherent non-rigidity and the tendency for a subject's gaze to follow the camera. Traditionally, high-quality 3D human scans required expensive "bullet-time" rigs with dozens of synchronized cameras. Nerfies addresses this by turning a standard smartphone into a high-fidelity 3D scanner for dynamic subjects.
Methods
The core of Nerfies is the decomposition of a dynamic scene into two components: a canonical NeRF template (representing the subject in a neutral state) and a volumetric deformation field.
- Volumetric Deformation Field: For every frame in a video, the system learns a unique deformation code. A Multi-Layer Perceptron (MLP) takes a 3D point from a specific video frame and "warps" it into the canonical coordinate system. To handle complex articulations better than simple translation, the authors use a dense SE(3) field, allowing the network to rotate and translate segments of the scene (like a head turning) naturally.
- Elastic Regularization: To ensure the deformation field remains physically plausible, the authors apply a penalty based on the Jacobian of the deformation. This treats the neural field like an elastic material that prefers to stay "as-rigid-as-possible," preventing unrealistic stretching of features like noses or limbs.
- Coarse-to-Fine Optimization: Coordinate-based networks are prone to local minima. The authors propose a strategy that starts by optimizing low-frequency structures and gradually introduces high-frequency details as training progresses. This ensures the global structure is correct before the network attempts to model fine textures like individual hairs.
Experimental Setup and Datasets
The method was evaluated using casual handheld selfie videos of humans and pets in various environments. To provide quantitative validation, the authors constructed a custom rig with two time-synchronized Pixel 3 phones. One phone's video was used for training, while the second phone, positioned 15cm away, provided "ground truth" images to test the model's ability to predict unseen viewpoints. Performance was measured using LPIPS (perceptual similarity), MS-SSIM, and PSNR across five distinct subjects.
Results and Key Insights
Nerfies significantly outperformed standard NeRF and existing dynamic baselines like Neural Volumes. Qualitatively, the method successfully reconstructs challenging thin structures such as individual hair strands, glasses, and clothing wrinkles that are typically lost in 3D reconstructions.
A notable discovery was the "Hollow-Face" gaze effect: if a subject's eyes consistently follow the camera during capture, the system interprets the moving gaze as part of the 3D geometry. This creates an illusion where the reconstructed model appears to maintain eye contact with the viewer from every angle. Additionally, the use of Jacobian-based regularization proved that physical plausibility can be enforced on a purely mathematical neural field without the need for a traditional 3D mesh.
Limitations and Future Work
While robust, the model faces challenges with major topological changes, such as a mouth opening and closing; the network often attempts to use view-dependent effects to hide these changes rather than updating the underlying geometry. Furthermore, the system relies on Structure-from-Motion (SfM) for camera tracking, which requires a static, feature-rich background. Future work aims to extend the framework to handle variable lighting and larger-scale full-body movements.
Conclusion
Nerfies brings professional-grade 3D human reconstruction to consumer devices, enabling a new form of personal media that is both accessible and photorealistic. By bridging the gap between static NeRFs and dynamic reality, it paves the way for free-viewpoint video and high-fidelity digital avatars captured with nothing more than a smartphone.
Summary updated