A Decoupled 3D Facial Shape Model by Adversarial Training

Victoria Fernández Abrevaya, Adnane Boukhayma, Stefanie Wuhrer, Edmond Boyer

Abstract

Data-driven generative 3D face models are used to compactly encode facial shape data into meaningful parametric representations. A desirable property of these models is their ability to effectively decouple natural sources of variation, in particular identity and expression. While factorized representations have been proposed for that purpose, they are still limited in the variability they can capture and may present modeling artifacts when applied to tasks such as expression transfer. In this work, we explore a new direction with Generative Adversarial Networks and show that they contribute to better face modeling performances, especially in decoupling natural factors, while also achieving more diverse samples. To train the model we introduce a novel architecture that combines a 3D generator with a 2D discriminator that leverages conventional CNNs, where the two components are bridged by a geometry mapping layer. We further present a training scheme, based on auxiliary classifiers, to explicitly disentangle identity and expression attributes. Through quantitative and qualitative results on standard face datasets, we illustrate the benefits of our model and demonstrate that it outperforms competing state of the art methods in terms of decoupling and diversity.

Introduction

In 3D computer vision, creating a "morphable model" of a human face is a foundational challenge. Ideally, such a model should be decoupled: a digital representation of a specific person should allow for the independent manipulation of expression without altering the underlying identity. For instance, applying a "smile" parameter should not inadvertently change the shape of the nose or the width of the jaw.

Traditional methods, such as Principal Component Analysis (PCA) or multilinear tensors, often struggle with this requirement. They frequently assume facial variations are linear—which they are not—or require massive, perfectly labeled datasets that are difficult to acquire. While recent deep learning approaches using autoencoders have improved geometric detail, they often "tangle" identity and expression together, leading to visual artifacts during expression transfer. This research introduces a non-linear, highly expressive GAN-based model designed to keep these factors strictly separated.

Methodology

Hybrid 3D-2D GAN Architecture

The authors propose a unique hybrid architecture that leverages the strengths of both 3D geometric modeling and 2D image processing.

  1. The Generator (3D): Unlike standard GANs that produce 2D images, this model uses a Multilayer Perceptron (MLP) to directly predict the 3D coordinates of a facial mesh. This direct generation avoids the "reconstruction artifacts" (such as jagged edges) that typically occur when attempting to convert a 2D "geometry image" back into a 3D shape.
  2. The Geometry Mapping Layer: This layer serves as the bridge between the 3D and 2D domains. It takes the generated 3D mesh and projects it into a 2D "geometry image" using UV parameterization. This transformation allows the model to utilize standard, high-performance 2D Convolutional Neural Networks (CNNs) for discrimination.
  3. The Discriminator (2D): A CNN evaluates the 2D geometry images. It is structured as an Auxiliary Classifier GAN (AC-GAN), meaning it does not just distinguish between "real" and "fake" samples; it also attempts to classify the identity and the expression of the face.
Overview of the proposed 3D-2D GAN architecture. The generator (G) produces 3D meshes which are then mapped to 2D UV space for evaluation by a convolutional discriminator (D) with auxiliary classifiers.

Training Objectives and Regularization

To ensure high-quality results and effective disentanglement, the model employs several specialized loss functions:

  • Feature Loss: To address sparse labeling (where identity is known but expression is not), a feature loss ensures that samples with the same "expression code" share similar internal feature representations in the discriminator, regardless of identity.
  • Mesh Regularization: A Laplacian loss is applied to the 3D output to ensure the generated surfaces remain smooth and realistic, preventing the noise often associated with raw coordinate generation.
Visual comparison of surface quality. The use of a direct 3D generator and mesh regularization prevents the jagged artifacts (middle columns) often found in 2D-to-3D reconstruction pipelines.

Experimental Setup

The model was trained on a comprehensive combined dataset of 30,559 registered 3D faces from four major sources: BU-3DFE, Bosphorus, BP4D-Spontaneous, and BU-4DFE. These sources provide a mix of high identity variability and rich expression sequences. A significant technical hurdle was the sparsity of labels; while identity labels were universal, expression labels were available for only approximately 7% of the data, making the "feature loss" approach critical for successful training.

Key Results

The model was benchmarked against industry standards like 3DMM and deep learning models such as CoMA and MAE.

  • Expression Decoupling: The proposed model achieved an expression decoupling score of 89.7%, significantly outperforming MAE (53.3%) and CoMA (65.5%).
  • Diversity: The GAN approach produced significantly more diverse facial shapes, achieving a Global Diversity of 4.74mm, whereas autoencoder-based methods tended to regress toward "average" faces.
  • Sparse Data Reconstruction: When fitting the model to just 85 landmarks, it achieved a lower Root Mean Square Error (2.42mm) than all competitors, demonstrating its strength as a prior for 3D reconstruction.

Conclusion

This research demonstrates that adversarial training, combined with a hybrid 3D-2D architecture, can effectively decouple identity and expression in facial modeling. While the model is highly effective, future work remains in accounting for temporal dynamics (animation) and extending the disentanglement to other factors such as age, gender, and speech-related visemes.

Summary updated

Abstract

Data-driven generative 3D face models are used to compactly encode facial shape data into meaningful parametric representations. A desirable property of these models is their ability to effectively decouple natural sources of variation, in particular identity and expression. While factorized representations have been proposed for that purpose, they are still limited in the variability they can capture and may present modeling artifacts when applied to tasks such as expression transfer. In this work, we explore a new direction with Generative Adversarial Networks and show that they contribute to better face modeling performances, especially in decoupling natural factors, while also achieving more diverse samples. To train the model we introduce a novel architecture that combines a 3D generator with a 2D discriminator that leverages conventional CNNs, where the two components are bridged by a geometry mapping layer. We further present a training scheme, based on auxiliary classifiers, to explicitly disentangle identity and expression attributes. Through quantitative and qualitative results on standard face datasets, we illustrate the benefits of our model and demonstrate that it outperforms competing state of the art methods in terms of decoupling and diversity.
BibTeX
@inproceedings{abrevaya2019decoupled,
  author = {Victoria Fernández Abrevaya and Adnane Boukhayma and Stefanie Wuhrer and Edmond Boyer},
  title = {{A Decoupled 3D Facial Shape Model by Adversarial Training}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=pXx3ARTj3y},
}
Figure