Information Bottlenecked Variational Autoencoder for Disentangled 3D Facial Expression Modelling

Hao Sun, Nick Pears, Yajie GuWACV 2022

TL;DR

This paper introduces a novel variational autoencoder (VAE) that effectively disentangles identity and expression from 3D facial scans. By employing an information bottleneck on the identity representation and a point cloud transformer as the encoder, the model achieves state-of-the-art results in disentanglement while maintaining competitive reconstruction quality.

What Problem Does This Paper Address?

Creating realistic and controllable 3D face models is a long-standing goal in computer vision. A key challenge is disentangling a person's unique identity from their facial expressions, allowing for independent manipulation of these attributes. Existing methods often struggle with this disentanglement, leading to models where identity and expression are intertwined, limiting their usefulness for applications like face reenactment or recognition. Furthermore, many approaches rely on mesh topology, which can be restrictive. This paper aims to address these limitations by proposing a more robust and flexible method for disentangled 3D facial expression modeling.

Methods

The core of this work is an Information Bottlenecked Variational Autoencoder (IB-VAE). The architecture consists of:

  • Point Cloud Transformer (PCT) Encoder: This novel encoder, based on attention mechanisms, processes unordered 3D point clouds directly, eliminating the need for explicit mesh topology. This makes the system more flexible and capable of handling diverse 3D face data.
  • Dual Decoders: Two decoders are employed: one reconstructs a neutral (identity-only) face, and the other reconstructs the full, expressive face.
  • Information Bottleneck on Identity: A crucial innovation is the application of an information bottleneck to the identity latent code. This is achieved through a mutual information regularizer that encourages the identity code to retain only identity-specific information, discarding expression-related details. This regularizer is formulated to jointly maximize the mutual information between the identity code and the reconstructed neutral face, while minimizing the mutual information between the input face and the identity code.
  • Conditional VAE (CVAE) Extension: For enhanced control and explainability, the authors extend their model to a CVAE. By incorporating expression labels, the CVAE can generate expressions with specific semantic meanings and varying intensities.

The overall loss function combines reconstruction losses (for both neutral and full faces), the KL divergence loss (standard in VAEs), and the proposed mutual information regularizer.

Datasets

The proposed method is evaluated on two publicly available 3D face datasets:

  • COMA Dataset [31]: Contains 3D scans of 12 individuals performing 12 different expressions, totaling 20,466 scans.
  • BU-3DFE [38]: Features 100 individuals with 6 different expressions across 4 intensity levels, resulting in 2,500 scans.

The paper notes that experiments are primarily based on point clouds, with mesh topology used only for visualization.

Key Insights and Ideas

  • Point Clouds are Sufficient: The success of the PCT encoder demonstrates that explicit mesh topology is not strictly necessary for effective 3D face modeling and disentanglement, opening doors for more flexible data handling.
  • Information Bottleneck for Disentanglement: Applying an information bottleneck specifically to the identity representation is a powerful technique for forcing the model to separate identity from expression, leading to significantly improved disentanglement.
  • Trade-off Between Disentanglement and Reconstruction: The paper highlights a fundamental trade-off: increasing the emphasis on disentanglement (via the information bottleneck) can lead to a slight decrease in reconstruction quality. However, the proposed method finds a good balance.
  • Conditional Generation for Explainability: The CVAE extension provides a clear pathway to generating semantically meaningful facial expressions by leveraging explicit labels, making the model more interpretable and controllable.

Key Results

  • State-of-the-Art Disentanglement: The proposed IB-VAE achieves state-of-the-art disentanglement error on both the COMA and BU-3DFE datasets, outperforming previous methods like Jiang et al. [19] and Zhang et al. [40]. For instance, on the COMA dataset, the disentanglement error is reduced to 0.005 mm.
  • Competitive Reconstruction: Despite the focus on disentanglement, the model maintains competitive reconstruction quality, with average vertex distances (AVD) comparable to or better than many existing methods. For example, on COMA, the AVD is 0.663 mm.
  • CVAE Performance: The CVAE extension on COMA achieved a reconstruction error of 0.740 mm and a disentanglement error of 0.008 mm, while enabling semantic expression generation.

Limitations and Future Work

  • Dataset Size and Registration: The performance on the BU-3DFE dataset is noted to be less ideal than on COMA. This is attributed to factors like less precise registration across subjects, smaller dataset size, and fewer expression samples per subject, which make disentanglement more challenging.
  • Hyperparameter Tuning: The authors acknowledge that finding the optimal balance for hyperparameter weights (like $\beta_{id}$ and $\beta_{mi}$) is critical and can be challenging due to the inherent trade-offs.
  • Variance in Disentanglement Error: The disentanglement error can exhibit higher variance, particularly when using sampling for mutual information estimation.
  • Future Work: The authors suggest exploring optimal weight selection for the information bottleneck and further improving the method's efficiency for datasets with fewer scans per subject. They also propose investigating how to find the optimal boundary for weight selection.

Summary updated

BibTeX
@inproceedings{sun2022information,
  author = {Hao Sun and Nick Pears and Yajie Gu},
  title = {{Information Bottlenecked Variational Autoencoder for Disentangled 3D Facial Expression Modelling}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure