Abstract
Data-driven generative 3D face models are used to compactly encode facial shape data into meaningful parametric representations. A desirable property of these models is their ability to effectively decouple natural sources of variation, in particular identity and expression. While factorized representations have been proposed for that purpose, they are still limited in the variability they can capture and may present modeling artifacts when applied to tasks such as expression transfer. In this work, we explore a new direction with Generative Adversarial Networks and show that they contribute to better face modeling performances, especially in decoupling natural factors, while also achieving more diverse samples. To train the model we introduce a novel architecture that combines a 3D generator with a 2D discriminator that leverages conventional CNNs, where the two components are bridged by a geometry mapping layer. We further present a training scheme, based on auxiliary classifiers, to explicitly disentangle identity and expression attributes. Through quantitative and qualitative results on standard face datasets, we illustrate the benefits of our model and demonstrate that it outperforms competing state of the art methods in terms of decoupling and diversity.
Introduction
In 3D computer vision, creating a "Morphable Model" (3DMM) of a human face is a foundational challenge. The goal is to develop a representation where identity (e.g., the unique structure of a person's face) and expression (e.g., a smile or a frown) are perfectly decoupled. This disentanglement allows for tasks like expression transfer, where one person's smile is mapped onto another person's identity.
Traditional linear models, such as PCA-based 3DMMs, struggle with the complex, non-linear deformations of skin and muscle. Consequently, swapping expressions often results in "ghosting" artifacts or unrealistic distortions. While deep learning autoencoders have improved these results, they typically require massive, perfectly labeled datasets that are difficult to acquire.
Methodology
This research introduces a novel Generative Adversarial Network (GAN) architecture designed specifically for 3D meshes, bridging the gap between 3D geometric precision and 2D convolutional power.
3D-2D Hybrid Architecture
The Generator (G) is a Multi-Layer Perceptron that outputs 3D vertex coordinates. To allow a standard 2D Discriminator (D) to evaluate these meshes, the authors introduce a Geometry Mapping Layer. This layer projects the 3D mesh into a 2D "geometry image" using UV coordinates. This allows the model to leverage powerful, standard Convolutional Neural Networks (CNNs) as the discriminator.
Disentanglement and Loss Functions
To ensure identity and expression remain separate, the architecture employs several specialized loss functions:
- Auxiliary Classifiers: The Discriminator predicts both the identity and the expression of the generated face, forcing the Generator to encode these attributes distinctly.
- Feature Loss: The model is trained on pairs of faces sharing one attribute (e.g., same person, different expression). By forcing internal mathematical representations to match for the shared attribute, the network learns to "lock" identity while varying expression.
- Mesh Regularization: A Laplacian loss is applied to ensure the generated 3D surfaces remain smooth and physically plausible, preventing "spiky" geometric artifacts.
Experimental Results
The model was trained on a combined dataset of over 30,000 registered 3D faces from sources including BU-3DFE, Bosphorus, and BP4D-Spontaneous. Despite only 7% of the data having expression labels, the model demonstrated high robustness.
- Superior Decoupling: In expression decoupling benchmarks, the proposed model achieved 89.7% accuracy, significantly outperforming previous state-of-the-art models which ranged between 53% and 65%.
- Increased Diversity: Unlike autoencoder-based methods that often suffer from "mode collapse" (generating average faces), the GAN approach produced a wide variety of unique identities and expressions.
- Landmark Reconstruction: When reconstructing 3D shapes from sparse 85-point landmarks, the model produced more accurate and realistic results than industry-standard 3DMM or CoMA models.
Conclusion
By combining 3D generation with 2D discrimination, this model achieves state-of-the-art performance in facial attribute decoupling and sample diversity. While currently focused on static frames, future work aims to address temporal consistency for video applications and expand the model to include additional factors such as age, gender, and fine-grained skin details.
Summary updated