Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks

Baris Gecer, Alexandros Lattas, Stylianos Ploumpis, Jiankang Deng, Athanasios Papaioannou, Stylianos Moschoglou, Stefanos Zafeiriou

TL;DR

This paper introduces TBGAN, a novel Generative Adversarial Network (GAN) architecture that jointly synthesizes high-quality 3D face texture, shape, and normals. This unified approach overcomes limitations of previous methods that treated these modalities separately, enabling more realistic and coherent 3D face generation, and can also be conditioned on facial expressions.

What Problem Does This Paper Address?

Generating realistic 3D faces is crucial for applications like computer graphics, gaming, and virtual reality. Traditional methods often rely on linear statistical models (like 3D Morphable Models or 3DMMs) which struggle to capture the fine details of facial texture and normals, leading to overly smoothed or unrealistic results. While recent GANs excel at generating high-quality 2D face images or textures, they often neglect the crucial interplay between shape, texture, and normals, or generate them independently. This disconnect can lead to inconsistencies, such as an aged texture on a young face geometry, hindering photorealism. The paper aims to bridge this gap by proposing a method that generates these coupled 3D modalities simultaneously, ensuring their correlations are maintained for a more faithful representation of human faces.

Methods

The core of the proposed method is a Trunk-Branch GAN (TBGAN) architecture. This design features:

  • Trunk-Branch Structure: A "trunk" network handles shared information and correlations across modalities, while separate "branch" networks specialize in generating texture, shape, and normals. This allows for modality-specific learning while preserving inter-modal relationships.
  • UV Space Representation: 3D face data (shape, texture, normals) is reparameterized into UV maps, which are then fed into the GAN. This image-like format is amenable to standard 2D convolutional operations.
  • Joint Modality Generation: The GAN is trained to generate all three modalities (texture, shape, normals) concurrently, ensuring their local and global correspondences are maintained.
  • Expression Conditioning: The model is extended to generate faces with specific expressions by conditioning the GAN on expression annotations, enabling control over facial emotions. This is achieved by incorporating expression encodings into the generator and discriminator.
  • WGAN-GP Loss: The authors utilize a Wasserstein GAN with Gradient Penalty (WGAN-GP) loss for stable training.

Datasets

  • MeIn3D Dataset: Used for training the core TBGAN model. This dataset contains approximately 10,000 neutral 3D facial scans with diverse age, gender, and ethnicity.
  • Science Museum Exhibition Data: A custom dataset of ~35,000 facial scans from ~5,000 individuals, captured with a 3DMD apparatus under various guided expressions. This dataset was used for training the expression-conditioned GAN.
  • CFP (Celebrities in Frontal Profile) Dataset: Used for evaluating pose-invariant face recognition.
  • AgeDB Dataset: Also used for evaluating pose-invariant face recognition, focusing on age variations.
  • CASIA Dataset: Used in conjunction with generated data for face recognition experiments.

Key Insights and Ideas

  • The Importance of Coupled Modalities: The paper strongly emphasizes that photorealistic 3D face synthesis requires the joint generation and correlation of texture, shape, and normals. Treating them independently leads to noticeable inconsistencies.
  • Trunk-Branch Architecture for Correlation: The proposed TBGAN effectively leverages a trunk-branch structure to capture both shared features and modality-specific details, a key innovation for generating coherent multi-modal 3D data.
  • UV Space as a Unifying Representation: Reparameterizing 3D faces into UV maps provides a consistent and efficient way to process and generate complex facial data with standard convolutional networks.
  • Expression Control Beyond Identity: The ability to condition generation on expression, while maintaining identity consistency, is a significant step towards more dynamic and expressive 3D avatars.

Key Results

  • Qualitative Results: The generated UV maps for shape, texture, and normals are shown to be highly correlated and coherent, leading to visually convincing 3D faces with realistic shading and detail. The paper demonstrates impressive results in generating diverse identities and expressions.
  • Quantitative Results (Face Recognition): When using generated faces as additional training data for a state-of-the-art face recognition model (ArcFace), verification performance on CFP-FP increased from 95.56% to 97.12%, a significant improvement. This highlights the utility of the generated data for downstream tasks.
  • Efficiency: The generator network requires only 1.25GB of memory, significantly less than a comparable PCA model (6GB). Generation time is reported to be a few milliseconds on a high-end GPU.

Limitations and Future Work

  • Dataset Dependence: The current model's capabilities are limited by the modalities present in the training data. The authors note that the framework can be extended to include more identity-specific modalities (e.g., cavity, gloss, scatter) if such data becomes available.
  • Rendering Improvements: While the generated data leads to photorealistic renderings, the paper acknowledges that further improvements can be made by incorporating more detailed identity-generic maps for reflectance properties.
  • Full Head Completion: The paper demonstrates extending the face generation to full head representations, suggesting this as a direction for future work to create more complete 3D avatars.
  • Generalization to Other Modalities: The trunk-branch architecture is designed to be extensible, hinting at its potential application to other types of 3D data beyond faces.

Summary updated

Abstract

Generating realistic 3D faces is of high importance for computer graphics and computer vision applications. Generally, research on 3D face generation revolves around linear statistical models of the facial surface. Nevertheless, these models cannot represent faithfully either the facial texture or the normals of the face, which are very crucial for photo-realistic face synthesis. Recently, it was demonstrated that Generative Adversarial Networks (GANs) can be used for generating high-quality textures of faces. Nevertheless, the generation process either omits the geometry and normals, or independent processes are used to produce 3D shape information. In this paper, we present the first methodology that generates high-quality texture, shape, and normals jointly, which can be used for photo-realistic synthesis. To do so, we propose a novel GAN that can generate data from different modalities while exploiting their correlations. Furthermore, we demonstrate how we can condition the generation on the expression and create faces with various facial expressions. The qualitative results shown in this paper are compressed due to size limitations, full-resolution results and the accompanying video can be found in the supplementary documents. The code and models are available at the project page: https://github.com/barisgecer/TBGAN .
BibTeX
@inproceedings{gecer2020synthesizing,
  author = {Baris Gecer and Alexandros Lattas and Stylianos Ploumpis and Jiankang Deng and Athanasios Papaioannou and Stylianos Moschoglou and Stefanos Zafeiriou},
  title = {{Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58526-6\_25},
}
Figure