Make a Face: Towards Arbitrary High Fidelity Face Manipulation

Shengju Qian, Kwan-Yee Lin, Wayne Wu, Yangxiaokang Liu, Quan Wang, Fumin Shen, Chen Qian, Ran He

Abstract

Face manipulation, the task of altering facial expressions or poses while preserving identity, is a significant challenge in computer vision. Traditional methods often rely on paired training data or struggle with the "mean face" problem, where generated images appear blurry due to simplistic latent space assumptions. We introduce AF-VAE (Additive Focal Variational Auto-encoder), a framework that achieves high-fidelity, diverse face manipulation without paired data. By incorporating an Additive Memory module to model a complex latent prior and employing Human Visual System (HVS)-aware architectural improvements, AF-VAE significantly outperforms existing GAN-based and VAE-based methods in both realism and diversity.

Introduction

The ability to manipulate facial attributes like expression and head pose is central to digital entertainment. However, two primary obstacles hinder progress: the scarcity of paired training data and the inherent limitations of standard Variational Auto-encoders (VAEs). While VAEs offer stable training compared to Generative Adversarial Networks (GANs), they often produce "average," blurry results because they assume a single Gaussian distribution for the latent space. This assumption fails to capture the multi-modal nature of human faces across various ages, complexions, and extreme poses.

Methods: The AF-VAE Framework

AF-VAE addresses these issues through three primary innovations within a Conditional VAE (C-VAE) structure.

Geometry-Guided Disentanglement

To separate identity (appearance) from expression and pose (structure), the model is conditioned on boundary maps derived from facial landmarks. This ensures the latent code focuses exclusively on texture and identity, while the conditioning signal dictates the geometric structure.

Additive Memory Encoding

The core innovation is the replacement of the standard Gaussian prior with an Additive Memory module. Using unsupervised clustering (K-means) on boundary maps, the researchers identify "focal points" in the latent space. The prior is modeled as an Additive Gaussian Mixture, allowing the latent space to represent distinct facial archetypes more effectively than a single distribution.

Figure 4: (a) The AF-VAE architecture overview showing the Additive Memory module and focal points. (b) Visualization of the clustered latent space.

HVS-Aware Architecture

To enhance perceptual quality, the model incorporates strategies inspired by the Human Visual System (HVS). Specifically, Sub-pixel Convolution (Pixel-Shuffle) replaces standard deconvolution to eliminate checkerboard artifacts. By "scattering" noise into high-frequency regions where the HVS is less sensitive, the perceived image quality improves significantly. Additionally, Weight Normalization (WN) is applied to stabilize training and increase sample diversity.

Results and Evaluation

The model was evaluated on datasets including CelebA, RaFD, and MultiPIE.

Quantitative Performance

On the CelebA dataset, AF-VAE achieved a Fréchet Inception Distance (FID) of 36.82, a massive improvement over the previous state-of-the-art of 71.3. The Inception Score (IS) also doubled from 1.065 to 2.15, indicating significantly higher diversity in generated samples. In user studies, AF-VAE's "Fool Rate" (the percentage of images perceived as real) was 36.4%, compared to just 7.0% for GANimation.

Qualitative Analysis

As shown in Figure 1, the model successfully handles a wide range of expressions and poses across different identities. It maintains high fidelity even during extreme 90-degree rotations and intense expressions (e.g., "fear" or "disgust"), where competing GAN-based models often produce ghosting or blur.

Figure 1: Qualitative results of AF-VAE showing diverse expression and pose manipulation across multiple subjects.

Conclusion

AF-VAE demonstrates that high-fidelity face manipulation is possible without paired data or adversarial training. By breaking the standard Gaussian assumption of VAEs and utilizing geometry-guided memory, the model avoids the "mean face" problem. While limitations remain regarding rare occlusions and identity preservation during extreme structural shifts, AF-VAE sets a new standard for stable, diverse, and realistic facial synthesis.

Summary updated

Abstract

Recent studies have shown remarkable success in face manipulation task with the advance of GANs and VAEs paradigms, but the outputs are sometimes limited to low-resolution and lack of diversity. In this work, we propose Additive Focal Variational Auto-encoder (AF-VAE), a novel approach that can arbitrarily manipulate high-resolution face images using a simple yet effective model and only weak supervision of reconstruction and KL divergence losses. First, a novel additive Gaussian Mixture assumption is introduced with an unsupervised clustering mechanism in the structural latent space, which endows better disentanglement and boosts multi-modal representation with external memory. Second, to improve the perceptual quality of synthesized results, two simple strategies in architecture design are further tailored and discussed on the behavior of Human Visual System (HVS) for the first time, allowing for fine control over the model complexity and sample quality. Human opinion studies and new state-of-the-art Inception Score (IS) / Frechet Inception Distance (FID) demonstrate the superiority of our approach over existing algorithms, advancing both the fidelity and extremity of face manipulation task.
BibTeX
@inproceedings{qian2019make,
  author = {Shengju Qian and Kwan-Yee Lin and Wayne Wu and Yangxiaokang Liu and Quan Wang and Fumin Shen and Chen Qian and Ran He},
  title = {{Make a Face: Towards Arbitrary High Fidelity Face Manipulation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.01013},
}
Figure