Relightable and Animatable Neural Avatars from Videos

Wenbin Lin, Chengwei Zheng, Jun-Hai Yong, Feng XuAAAI 2024 · DOI

TL;DR

This paper introduces a novel method for creating relightable and animatable 3D human avatars from sparse videos, even under challenging, uncalibrated lighting. Their key innovation is an "invertible deformation field" that accurately reconstructs dynamic body geometry and enables physically plausible shadow effects, a significant step towards realistic virtual humans.

What Problem Does This Paper Address?

Creating realistic 3D digital avatars of humans that can be posed and relit under arbitrary lighting conditions is a long-standing challenge. Existing methods often require extensive, high-quality capture setups (like light stages) or struggle to disentangle geometry, material properties, and lighting, especially when the human subject is in motion. This motion introduces complex shadow changes that are difficult to model. While Neural Radiance Fields (NeRFs) have shown promise for rendering, adapting them for dynamic, relightable human avatars with accurate shadows from everyday videos remains a significant hurdle.

Methods

The authors propose a three-stage pipeline:

  1. Geometry and Motion Reconstruction: They model dynamic body deformation using a combination of articulated rigid motion (via SMPL model) and a novel invertible neural deformation field. This field creates a bidirectional mapping between a canonical pose and observed poses, allowing for accurate inverse skinning and high-quality geometry reconstruction. This is a key departure from previous methods that used unidirectional deformation fields.
  2. Part-wise Light Visibility Estimation: To accurately model shadows, they introduce a pose-aware part-wise light visibility network. Instead of a single complex network, they break down the problem into smaller, part-specific networks. This approach simplifies the estimation of how light is blocked by different body parts under various poses and lighting conditions, leading to better shadow generation.
  3. Material and Light Estimation: With the geometry and visibility fixed, they optimize material properties (using the Disney BRDF model) and environment lighting parameters. They use a differentiable renderer to reconstruct the scene and minimize photometric losses. Regularization techniques are employed to address ambiguities in material and lighting disentanglement.

Datasets

  • Synthetic Dataset: Created by rigging characters and transferring motion from the ZJU-MoCap dataset, rendered with multi-view videos under different illuminations and OLAT light sources.
  • Real Datasets: ZJU-MoCap, Human3.6M, PeopleSnapshot, and Mixamo (for motion).

Key Insights and Ideas

  • Invertible Deformation Field for Geometry: The core idea is that an invertible field allows for accurate bidirectional mapping between canonical and observed poses. This is crucial for solving the inverse skinning problem more robustly than previous methods, leading to better geometry.
  • Part-wise Visibility for Shadows: Decomposing the complex light visibility problem into smaller, part-specific networks significantly improves the ability to model dynamic self-occlusion and generate physically plausible shadows.
  • Disentanglement is Key: The entire pipeline is built around the principle of disentangling geometry, material, and lighting, which is essential for achieving relightability and animatability.

Key Results

  • Quantitative: The method achieves state-of-the-art results in relighting and geometry reconstruction compared to methods like Relighting4D, ARAH, and Peng et al. (2022), as shown in Tables 1 and 2. For instance, their method achieves better PSNR, SSIM, and LPIPS scores for relighting.
  • Qualitative: The paper demonstrates high-quality relighting results with realistic shadows on both synthetic and real data (Figures 2, 3, and 5). The method successfully synthesizes plausible self-occlusions and handles novel poses better than existing approaches.

Limitations and Future Work

  • Training Time: The network training is computationally intensive, taking about 2.5 days on a single RTX 3090 GPU. Inference is faster at around 40 seconds per image.
  • Animation Complexity: The method struggles with animating pose-dependent wrinkle deformations (especially for loose clothing) and generating global illumination effects.
  • Focus: The current work focuses on body motion and does not explicitly handle facial expressions or hand movements, though recent works are starting to address these.
  • Future Directions: The authors suggest exploring instant training techniques (like Instant-NGP) to improve efficiency and extending the method to handle more complex clothing dynamics and facial/hand animations.

Summary updated

BibTeX
@inproceedings{lin2024relightable,
  author = {Wenbin Lin and Chengwei Zheng and Jun-Hai Yong and Feng Xu},
  title = {{Relightable and Animatable Neural Avatars from Videos}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i4.28136},
}
Figure