InvertAvatar: Incremental GAN Inversion for Generalized Head Avatars

Xiaochen Zhao, Jingxiang Sun, Lizhen Wang, Jinli Suo, Yebin Liu

TL;DR

This paper introduces InvertAvatar, a novel framework for creating high-fidelity, animatable 3D head avatars from single or multiple images. It addresses limitations in existing methods by leveraging an incremental GAN inversion approach with recurrent networks to effectively fuse information from multiple frames, leading to improved reconstruction quality and expression control.

What Problem Does This Paper Address?

Creating realistic and controllable 3D digital head avatars is crucial for applications like AR/VR and telepresence. However, current methods often struggle with issues like shape distortion, inaccurate expressions, and identity flickering, especially when dealing with limited input data. Existing one-shot inversion techniques, which aim to reconstruct an avatar from a single image, fail to fully utilize the richer information available in multiple frames. This limits their ability to capture fine-grained details and achieve accurate reconstructions, particularly for complex expressions and poses.

Methods

InvertAvatar builds upon an animatable 3D GAN prior, specifically Next3D, but introduces key modifications. The core innovation is Incremental 3D GAN Inversion, which allows for the progressive refinement of avatar quality as more frames are processed.

Key techniques include:

  • Unique Animatable 3D GAN Prior: The authors propose a more compact and memory-efficient GAN architecture that enhances expression controllability. This involves a novel Gface module conditioned on multi-scale rasterized neural textures and a deformable mesh model, improving the handling of inner mouth features.
  • UV-Aligned Neural Texture Encoder: A specialized neural texture encoder is introduced that categorizes texture features based on UV parameterization. This enables pixel-aligned image-to-image translation, which is crucial for accurately mapping observed image features to the canonical texture space without needing to learn complex correspondences.
  • ConvGRU-based Recurrent Networks: To effectively aggregate temporal information from multiple input frames (especially in video sequences), the framework incorporates ConvGRU-based recurrent networks (Etex_rec and Etri_rec). These networks allow for a flexible number of inputs and intelligently learn which information to retain or discard, leading to improved geometry and texture detail reconstruction.

The inversion process is a two-stage approach: a coarse stage for initial latent space inversion and a fine stage for refining features in canonical spaces using the novel encoders and recurrent networks.

Datasets

The paper uses the following datasets:

  • FFHQ: Used for training the animatable 3D GAN prior and for training encoders.
  • VFHQ: Used for training the temporal aggregation networks (ConvGRU-based recurrent networks) with real-world video clips.
  • CelebA-HQ: Used for evaluation, particularly for cross-identity reenactment.
  • HDTF: Used for evaluation, including same-ID and cross-ID reenactment.

Key Insights and Ideas

  • Incremental Refinement is Key: The core insight is that by incrementally processing frames and refining the avatar, the system can achieve significantly higher fidelity than a single-shot approach, especially for complex details and expressions.
  • UV Parameterization for Texture: Leveraging UV parameterization for texture features allows for a more direct and effective mapping from image observations to the canonical texture space, simplifying the learning process and improving detail recovery.
  • Recurrent Networks for Temporal Fusion: ConvGRU proves to be a robust choice for temporal aggregation, outperforming fixed-window baselines by adaptively managing information over time, which is crucial for handling streaming video data.
  • Compact GAN for Efficiency: A more efficient GAN architecture, while slightly increasing FID, leads to better expression control and reduced memory footprint, demonstrating a favorable trade-off for avatar animation tasks.

Key Results

  • State-of-the-Art Performance: InvertAvatar achieves state-of-the-art results on one-shot and few-shot avatar animation tasks, outperforming existing methods in terms of realistic appearance recovery, identity preservation, and expression control.
  • Improved Fidelity with More Frames: Quantitative evaluations (e.g., Table 3) show that reconstruction quality (PSNR, LPIPS, CSIM) improves as the number of input frames increases.
  • Efficiency: The method reconstructs avatars in under a second, with only marginal increases in time cost for additional input frames.
  • Qualitative Improvements: Figures demonstrate superior results in capturing fine details like hair strands, earrings, and nuanced facial expressions (e.g., asymmetrical mouth movements, eye gazes) compared to baselines.
  • Reduced Parameter Count: The proposed animatable 3D GAN prior has roughly half the parameters of Next3D while achieving comparable or better performance in animation accuracy.

Limitations and Future Work

  • Parametric Model Dependency: The method's reliance on a parametric model for expression control can lead to issues with extreme expressions beyond the model's capabilities (e.g., frowning, sticking tongue out).
  • Dynamic Lower Teeth: Synthesizing stable and clear dynamic lower teeth remains a challenge.
  • Future Work: The authors suggest that more robust facial models capable of tracking a wider range of expressions could further enhance performance. They also highlight the potential for their method to be used for generating "deepfakes," emphasizing the need for careful consideration before deployment.

Summary updated

Abstract

While high fidelity and efficiency are central to the creation of digital head avatars, recent methods relying on 2D or 3D generative models often experience limitations such as shape distortion, expression inaccuracy, and identity flickering. Additionally, existing one-shot inversion techniques fail to fully leverage multiple input images for detailed feature extraction. We propose a novel framework, \textbf{Incremental 3D GAN Inversion}, that enhances avatar reconstruction performance using an algorithm designed to increase the fidelity from multiple frames, resulting in improved reconstruction quality proportional to frame count. Our method introduces a unique animatable 3D GAN prior with two crucial modifications for enhanced expression controllability alongside an innovative neural texture encoder that categorizes texture feature spaces based on UV parameterization. Differentiating from traditional techniques, our architecture emphasizes pixel-aligned image-to-image translation, mitigating the need to learn correspondences between observation and canonical spaces. Furthermore, we incorporate ConvGRU-based recurrent networks for temporal data aggregation from multiple frames, boosting geometry and texture detail reconstruction. The proposed paradigm demonstrates state-of-the-art performance on one-shot and few-shot avatar animation tasks. Code will be available at https://github.com/XChenZ/invertAvatar.
BibTeX
@inproceedings{zhao2024invertavatar,
  author = {Xiaochen Zhao and Jingxiang Sun and Lizhen Wang and Jinli Suo and Yebin Liu},
  title = {{InvertAvatar: Incremental GAN Inversion for Generalized Head Avatars}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2024},
  doi = {10.1145/3641519.3657478},
}

Loading related content...

Figure