AvatarMMC: 3D Head Avatar Generation and Editing with Multi-Modal Conditioning

Wamiq Reyaz Para, Abdelrahman Eldesokey, Zhenyu Li, Pradyumna Reddy, Jiankang Deng, Peter WonkaarXiv 2024 · OpenReview

Abstract

We introduce AvatarMMC, an approach for 3D head avatar generation and editing with multi-modal conditioning based on a 3D Generative Adversarial Network (GAN) and a Latent Diffusion Model (LDM). While 3D GANs produce high-quality avatars, controlling them with multiple modalities remains challenging. We propose to exploit the conditioning capabilities of LDMs to enable multi-modal control over the latent space of a pre-trained 3D GAN. Our method generates and edits 3D head avatars given a mixture of control signals such as RGB input, segmentation masks, and global attributes. Experiments demonstrate that AvatarMMC outperforms solely GAN-based methods qualitatively and quantitatively in both generation and editing tasks.

Introduction

The creation of realistic 3D head avatars is essential for gaming, virtual reality, and film. However, achieving fine-grained control over these avatars is a persistent challenge. Current 3D Generative Adversarial Networks (GANs) generate high-fidelity images but are difficult to steer, often requiring slow, complex optimization. Conversely, 2D Diffusion Models offer excellent multi-modal controllability but struggle with 3D consistency across varying camera angles. AvatarMMC bridges this gap by using a diffusion model to navigate the latent space of a pre-trained 3D GAN, enabling precise control without the computational cost of diffusing high-dimensional 3D representations.

Methodology

AvatarMMC treats a pre-trained 3D GAN (Next3D) as a frozen rendering engine and utilizes a 1D Latent Diffusion Model (LDM) as a navigator within the GAN's latent space.

Latent Space Navigation

Instead of diffusing pixels or 3D tri-planes, the model operates on the GAN’s StyleSpace ($\mathcal{S}$), a sequence of 1D vectors controlling facial features. This reduces dimensionality by approximately 50x compared to direct 3D representation diffusion, inheriting the GAN's 3D consistency for free. The resulting UNet is highly efficient, containing only 225M parameters.

Multi-modal conditioning example combining visual and attribute inputs.

Multi-Modal Conditioning

The framework employs specialized encoders for diverse inputs:

  • Attribute Encoder: Processes text-based traits (e.g., "young," "brown hair") using sinusoidal embeddings and an MLP.
  • Visual Encoder: Simultaneously processes RGB images and facial segmentation masks using a modified DeepLabV3+.

For editing, a two-stage sampling process is used: a reconstruction stage to find the original face's latent code, followed by an editing stage where specific regions are masked to allow the diffusion model to "inpaint" new features while preserving identity.

Dataset Generation

To train the model, the authors generated a synthetic dataset of 6 million unique faces using the Next3D GAN. These faces were automatically labeled using the FaRL network for attributes and a segmentation network for masks. This self-supervised approach provided a perfectly labeled, 3D-consistent training set for the diffusion model.

Results and Evaluation

AvatarMMC demonstrates significant improvements in both speed and quality. The model generates a latent code in just 0.25 seconds, compared to 90 seconds for optimization-based baselines like $W^{++}$.

High-fidelity 3D head avatar generated by the AvatarMMC framework.

Quantitatively, AvatarMMC achieved a PSNR of 22.18 (vs. 19.38 for the baseline) and an identity preservation score of 0.70 (vs. 0.48). Qualitatively, the model maintains identity more effectively when adding accessories or changing attributes compared to GAN-only approaches.

Conclusion

AvatarMMC is the first approach to introduce multi-modal conditioning to 3D avatar generation and editing. By diffusing 1D latent codes, it achieves high-fidelity, 3D-consistent results with unprecedented efficiency and control. Future work will explore broader conditioning modalities and direct expression control.

Summary updated

Abstract

We introduce an approach for 3D head avatar generation and editing with multi-modal conditioning based on a 3D Generative Adversarial Network (GAN) and a Latent Diffusion Model (LDM). 3D GANs can generate high-quality head avatars given a single or no condition. However, it is challenging to generate samples that adhere to multiple conditions of different modalities. On the other hand, LDMs excel at learning complex conditional distributions. To this end, we propose to exploit the conditioning capabilities of LDMs to enable multi-modal control over the latent space of a pre-trained 3D GAN. Our method can generate and edit 3D head avatars given a mixture of control signals such as RGB input, segmentation masks, and global attributes. This provides better control over the generation and editing of synthetic avatars both globally and locally. Experiments show that our proposed approach outperforms a solely GAN-based approach both qualitatively and quantitatively on generation and editing tasks. To the best of our knowledge, our approach is the first to introduce multi-modal conditioning to 3D avatar generation and editing. \\href{avatarmmc-sig24.github.io}{Project Page}
BibTeX
@misc{para2024avatarmmc,
  author = {Wamiq Reyaz Para and Abdelrahman Eldesokey and Zhenyu Li and Pradyumna Reddy and Jiankang Deng and Peter Wonka},
  title = {{AvatarMMC: 3D Head Avatar Generation and Editing with Multi-Modal Conditioning}},
  year = {2024},
  eprint = {2402.05803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.05803},
}
Figure