Abstract
Face manipulation, the task of altering facial expressions or poses while preserving identity, is a significant challenge in computer vision. Traditional methods often rely on paired training data or struggle with the "mean face" problem, where generated images appear blurry due to simplistic latent space assumptions. We introduce AF-VAE (Additive Focal Variational Auto-encoder), a framework that achieves high-fidelity, diverse face manipulation without paired data. By incorporating an Additive Memory module to model a complex latent prior and employing Human Visual System (HVS)-aware architectural improvements, AF-VAE significantly outperforms existing GAN-based and VAE-based methods in both realism and diversity.
Introduction
The ability to manipulate facial attributes like expression and head pose is central to digital entertainment. However, two primary obstacles hinder progress: the scarcity of paired training data and the inherent limitations of standard Variational Auto-encoders (VAEs). While VAEs offer stable training compared to Generative Adversarial Networks (GANs), they often produce "average," blurry results because they assume a single Gaussian distribution for the latent space. This assumption fails to capture the multi-modal nature of human faces across various ages, complexions, and extreme poses.
Methods: The AF-VAE Framework
AF-VAE addresses these issues through three primary innovations within a Conditional VAE (C-VAE) structure.
Geometry-Guided Disentanglement
To separate identity (appearance) from expression and pose (structure), the model is conditioned on boundary maps derived from facial landmarks. This ensures the latent code focuses exclusively on texture and identity, while the conditioning signal dictates the geometric structure.
Additive Memory Encoding
The core innovation is the replacement of the standard Gaussian prior with an Additive Memory module. Using unsupervised clustering (K-means) on boundary maps, the researchers identify "focal points" in the latent space. The prior is modeled as an Additive Gaussian Mixture, allowing the latent space to represent distinct facial archetypes more effectively than a single distribution.
HVS-Aware Architecture
To enhance perceptual quality, the model incorporates strategies inspired by the Human Visual System (HVS). Specifically, Sub-pixel Convolution (Pixel-Shuffle) replaces standard deconvolution to eliminate checkerboard artifacts. By "scattering" noise into high-frequency regions where the HVS is less sensitive, the perceived image quality improves significantly. Additionally, Weight Normalization (WN) is applied to stabilize training and increase sample diversity.
Results and Evaluation
The model was evaluated on datasets including CelebA, RaFD, and MultiPIE.
Quantitative Performance
On the CelebA dataset, AF-VAE achieved a Fréchet Inception Distance (FID) of 36.82, a massive improvement over the previous state-of-the-art of 71.3. The Inception Score (IS) also doubled from 1.065 to 2.15, indicating significantly higher diversity in generated samples. In user studies, AF-VAE's "Fool Rate" (the percentage of images perceived as real) was 36.4%, compared to just 7.0% for GANimation.
Qualitative Analysis
As shown in Figure 1, the model successfully handles a wide range of expressions and poses across different identities. It maintains high fidelity even during extreme 90-degree rotations and intense expressions (e.g., "fear" or "disgust"), where competing GAN-based models often produce ghosting or blur.
Conclusion
AF-VAE demonstrates that high-fidelity face manipulation is possible without paired data or adversarial training. By breaking the standard Gaussian assumption of VAEs and utilizing geometry-guided memory, the model avoids the "mean face" problem. While limitations remain regarding rare occlusions and identity preservation during extreme structural shifts, AF-VAE sets a new standard for stable, diverse, and realistic facial synthesis.
Summary updated