Abstract
We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject — including both body and clothing — and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.
Introduction
Neural Radiance Fields (NeRF) have significantly advanced novel-view synthesis for static scenes. However, applying NeRF to dynamic human subjects "in the wild" presents two major challenges: data scarcity and geometric complexity. Unlike static objects, moving humans are typically captured from very few viewpoints at any given moment. Furthermore, humans exhibit complex deformations from clothing and hair that standard rigid NeRF formulations cannot easily capture. H-NeRF addresses these issues by unifying volumetric rendering with statistical human body priors, arguing that high-quality rendering and accurate 3D reconstruction are interdependent.
Methodology
The core of H-NeRF is the integration of imGHUM, a statistical implicit human body model, into the NeRF framework to provide a structured geometric prior.
Canonical Warping and Residual SDF
To handle temporal dynamics, H-NeRF maps 3D points from individual video frames back to a "canonical" reference frame using the imGHUM model. This allows the system to aggregate observations across time. Since standard body models only represent skin, H-NeRF introduces a Residual Signed Distance Function (SDF). This trainable layer captures non-rigid details such as clothing folds and hairstyles.
SDF-Guided NeRF
The model couples the NeRF volume density ($\sigma$) with the SDF ($d$) to ensure geometric consistency. By calculating a "pseudo-alpha" occupancy value from the SDF, the system regularizes the NeRF, preventing the "cloudy" artifacts common in sparse-view reconstructions. Both appearance and geometry are conditioned on body pose ($\theta$) and root transformation ($T$), enabling the simulation of pose-dependent clothing deformations and lighting shifts.
Experimental Results
H-NeRF was evaluated on synthetic datasets (RenderPeople, GHS3D) and real-world data (PeopleSnapshot, Human3.6M).
Performance and Generalization
Quantitative results on the RenderPeople dataset show H-NeRF achieving a PSNR of 28.78, outperforming NeuralBody (27.33). Geometric accuracy, measured by Chamfer distance, was more than twice as good as competing methods. Qualitatively, H-NeRF produces sharper details and maintains structural integrity even when rendering poses or angles not seen during training. The model's reliance on a statistical prior allows for "statistical extrapolation," generating plausible renders for entirely new poses.
Conclusion
H-NeRF demonstrates that human body priors can serve as powerful regularizers for neural radiance fields, enabling robust 4D reconstruction from sparse data. While currently limited to single-subject scenes with clean foreground masks and sensitivity to initial pose estimation, it paves the way for high-fidelity "in-the-wild" performance capture. Future work includes handling multiple interacting subjects and incorporating multi-modal cues.
Summary updated