H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion

Hongyi Xu, Thiemo Alldieck, Cristian Sminchisescu

Overview

H-NeRF (Human Neural Radiance Fields) is a novel framework that combines the photorealistic rendering capabilities of Neural Radiance Fields (NeRF) with the structural intelligence of a statistical human body model. By integrating an implicit human prior (imGHUM), the system can reconstruct and render humans in motion with high fidelity, even when provided with extremely sparse camera views or a single monocular video.

The Challenge of Free-Viewpoint Video

Creating "free-viewpoint" video—the ability to move a virtual camera around a person in motion—is a holy grail for AR, VR, and telepresence. However, two major challenges persist:

  1. Data Scarcity: Most high-end volumetric capture systems require dozens of synchronized cameras. In "in-the-wild" scenarios, we often only have a few views or a single moving camera.
  2. Geometric vs. Appearance Trade-offs: Traditional 3D reconstruction focuses on geometry but often produces "uncanny" or low-quality textures. Conversely, NeRF produces stunning images but struggles with dynamic scenes and sparse data, often resulting in "cloudy" artifacts or failing to generalize to poses not seen during training.

H-NeRF bridges this gap by using a human-specific geometric prior to "guide" the radiance field, ensuring the result is both photorealistic and anatomically plausible.

Methods

The core innovation of H-NeRF is the fusion of a Neural Radiance Field with an Implicit Statistical Human Body Model (imGHUM).

H-NeRF Architecture
  • Canonical Mapping: Instead of learning a radiance field for every frame, H-NeRF warps all 3D points from a specific video frame into a "canonical" (standardized) reference space using the imGHUM model. This allows the network to aggregate information from the entire video into a single, consistent representation.
  • Residual SDF: To capture details that a standard body model misses—like hair, loose clothing, and shoes—the authors learn a Residual Signed Distance Function (SDF). This "delta" is added to the base human model to create a personalized, high-detail geometry.
  • Joint Learning & Coupling Losses: The system co-trains the radiance field and the SDF. A novel "pseudo-alpha" loss uses the SDF values to regularize the NeRF's opacity. Essentially, it tells the NeRF: "If the SDF says this point is deep inside the body or far away in empty space, the opacity should be 1 or 0 accordingly."
  • Pose-Dependent Conditioning: To handle how clothing folds or how light hits the body as it moves, the NeRF and SDF networks are conditioned on the person's skeletal pose and root transformation.

Datasets

The authors evaluated H-NeRF on a mix of synthetic and real-world data:

  • RenderPeople & GHS3D: Synthetic datasets featuring animated 3D human scans, providing ground-truth geometry for precise quantitative evaluation.
  • PeopleSnapshot: A real-world dataset of monocular videos where a person rotates in front of a camera.
  • Human3.6M: A widely used multi-view dataset of humans performing various activities.

Key Insights and Ideas

Geometry Comparison
  • Structure as a Regularizer: Standard NeRF is "geometry-agnostic." By forcing the radiance field to live within the constraints of a human SDF, H-NeRF eliminates the "floaters" and "fog" typically seen when NeRF is trained on sparse views.
  • Statistical Extrapolation: Because the model is anchored to imGHUM (which is trained on thousands of human scans), it can perform "statistical extrapolation." You can render the subject in a completely new pose—like a high kick—even if they only walked during the training video.
  • Analysis-by-Synthesis Fine-tuning: The authors don't just take the initial pose estimation as gospel. They fine-tune the body shape and pose parameters during the NeRF training process to ensure a perfect alignment between the 3D model and the 2D image pixels.

Key Results

  • Superior Performance: H-NeRF significantly outperformed state-of-the-art methods like NeuralBody across all metrics. On the RenderPeople dataset, it achieved a PSNR of 28.78 compared to NeuralBody's 27.33.
  • Geometric Accuracy: The geometric reconstruction was much tighter, with a Chamfer distance (error) roughly 2.5x lower than competing methods.
  • Data Efficiency: The system is remarkably robust to sparse data. It can produce high-quality reconstructions using as few as 10 temporal frames from a four-camera setup.
  • Generalization: Qualitatively, H-NeRF produces much sharper textures and more detailed clothing folds, especially when viewed from angles not present in the training set.

Limitations and Future Work

While H-NeRF is a major step forward, the authors note several limitations. The current implementation assumes a single person in the scene and works best with "everyday" clothing. Extremely loose garments (like long dresses or capes) or handheld props might break the canonical warping logic. Furthermore, the quality still depends on a reasonably good initial estimate of the person's pose and a clean foreground mask. Future directions include integrating audio-visual models or extending the framework to handle multiple interacting people.

Summary updated

Abstract

We present neural radiance fields for rendering and temporal (4D) reconstruction of humans in motion (H-NeRF), as captured by a sparse set of cameras or even from a monocular video. Our approach combines ideas from neural scene representation, novel-view synthesis, and implicit statistical geometric human representations, coupled using novel loss functions. Instead of learning a radiance field with a uniform occupancy prior, we constrain it by a structured implicit human body model, represented using signed distance functions. This allows us to robustly fuse information from sparse views and generalize well beyond the poses or views observed in training. Moreover, we apply geometric constraints to co-learn the structure of the observed subject -- including both body and clothing -- and to regularize the radiance field to geometrically plausible solutions. Extensive experiments on multiple datasets demonstrate the robustness and the accuracy of our approach, its generalization capabilities significantly outside a small training set of poses and views, and statistical extrapolation beyond the observed shape.
BibTeX
@inproceedings{xu2021h,
  author = {Hongyi Xu and Thiemo Alldieck and Cristian Sminchisescu},
  title = {{H-NeRF: Neural Radiance Fields for Rendering and Temporal Reconstruction of Humans in Motion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=s-NI4H4e3Rf},
}
Figure