Neural Reconstruction of Relightable Human Model from Monocular Video

Wenzhang Sun, Yunlong Che, Yandong Guo, Han HuangICCV 2023 · DOI

Abstract

This paper introduces a self-supervised framework capable of reconstructing a fully relightable and animatable 3D human model from a single monocular video. By decomposing human appearance into geometry and spatially varying material properties (BRDF) within a canonical space, the model allows for high-quality rendering under novel poses and arbitrary lighting conditions. The approach overcomes the limitations of traditional Neural Radiance Fields (NeRF) which often "bake" lighting into the model, preventing realistic relighting in new environments.

Introduction

Creating realistic digital humans typically requires expensive multi-view camera rigs or specialized "Light Stages." While NeRF-based methods have simplified modeling from sparse setups, they generally fail to separate intrinsic material properties from the environment's lighting. This challenge is amplified when using monocular (single-camera) video, where one must simultaneously solve for 3D shape, dynamic deformation, material properties (e.g., shininess vs. matteness), and unknown scene illumination from a single 2D viewpoint.

Overview of the proposed framework, showing the transition from monocular input video to a decomposed 3D model capable of free-view rendering and relighting.

Methodology

The proposed framework separates human appearance from motion through a structured pipeline:

  1. Human Appearance Field: The model learns Multi-Layer Perceptrons (MLPs) in a canonical "T-pose" space. These MLPs output density, surface normals, and Disney BRDF parameters (base color, roughness, and metalness).
  2. Pose-driven Deformation Field: A bidirectional mapping bridges the gap between video frames and the canonical space. Forward Mapping warps points from the observed frame to the canonical space for geometry determination, while Inverse Mapping warps canonical attributes back to the observation space. This is critical for ensuring surface normals rotate correctly with body movement.
  3. Physically-Based Rendering (PBR): Final images are generated by combining learned materials and normals with a learned environment map represented by Spherical Gaussians.
  4. Progressive Training: A coarse-to-fine strategy is employed, starting with basic geometry and gradually introducing BRDF and lighting optimization to avoid local minima.
The detailed system architecture, highlighting the forward and inverse mapping processes and the integration of physically-based rendering.

Key Insights

  • Dynamic Normals: Accurate relighting depends on surface normals that respond to both skeletal rotation and non-rigid deformations (like clothing folds).
  • Sparsity and Smoothness Priors: To resolve ambiguities in monocular data, the authors use a Sparsity Prior to ensure base colors remain "clean" and free from baked-in shadows.
  • Decoupled Warping: Decoupling forward and inverse warping allows the model to accurately query canonical attributes while maintaining correct view-dependent effects, such as highlights on moving garments.

Results and Evaluation

The method was evaluated using the ZJU-Mocap dataset (simulated as monocular) and a custom Synthetic Dataset created in Blender for ground-truth comparison.

  • Quantitative Performance: On synthetic data, the method achieved a PSNR of 29.87, significantly outperforming Relighting4D (27.57). It also demonstrated superior accuracy in surface normal estimation.
  • Qualitative Quality: Reconstructions show realistic self-shadowing and consistent material appearance across complex motions.
  • Animation Capability: Linked to the SMPL body model, the reconstructed characters can be driven by entirely new motion sequences, such as applying a dance routine to a person captured while walking.

Conclusion

This work demonstrates a significant step forward in creating animatable, relightable digital humans from accessible monocular video. While challenges remain regarding high-frequency details (like hair) and extreme pose occlusions, the decomposition of appearance into intrinsic BRDF properties provides a robust foundation for realistic human synthesis in arbitrary virtual environments.

Summary updated

BibTeX
@inproceedings{sun2023neural,
  author = {Wenzhang Sun and Yunlong Che and Yandong Guo and Han Huang},
  title = {{Neural Reconstruction of Relightable Human Model from Monocular Video}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00043},
}
Figure