Mask-aware Photorealistic Face Attribute Manipulation

Ruoqi Sun, Chen Huang, Jianping Shi, Lizhuang Ma

Abstract

The task of face attribute manipulation has found increasing applications, but still remains challenging with the requirement of editing the attributes of a face image while preserving its unique details. In this paper, we choose to combine the Variational AutoEncoder (VAE) and Generative Adversarial Network (GAN) for photorealistic image generation. We propose an effective method to modify a modest amount of pixels in the feature maps of an encoder, changing the attribute strength continuously without hindering global information. Our training objectives of VAE and GAN are reinforced by the supervision of face recognition loss and cycle consistency loss for faithful preservation of face details. Moreover, we generate facial masks to enforce background consistency, which allows our training to focus on manipulating the foreground face rather than background. Experimental results demonstrate our method, called Mask-Adversarial AutoEncoder (M-AAE), can generate high-quality images with changing attributes and outperforms prior methods in detail preservation.

Introduction

Editing facial attributes—such as changing a smile to a frown or altering hair color—remains a significant challenge in computer vision. While Generative Adversarial Networks (GANs) have achieved high photorealism, they often suffer from identity loss, where the edited person no longer resembles the original, and background "ghosting," where the background is inadvertently modified. Previous methods typically employed global attribute vectors that shifted the entire image's tone. This work introduces the Mask-Adversarial AutoEncoder (M-AAE) to decouple attribute manipulation from the rest of the image, ensuring that facial identity and background remain intact.

Methodology

The M-AAE framework is built upon a VAE-GAN hybrid architecture. The VAE provides a structured latent space for reconstruction, while the GAN ensures high-frequency photorealistic details.

M-AAE Framework Overview: The architecture combines VAE and GAN components with five distinct loss functions to ensure identity and background preservation.

Sparse Latent Modification

A key innovation is the use of sparse feature modification. Instead of applying a global vector to the entire latent representation, the model identifies a minimal number of pixels in the encoder's feature maps. By modifying only these sparse points by a scalar value $\delta$, the system can control attribute strength continuously without distorting global information or causing color bleeding.

Training Objectives

The model is optimized using a five-loss objective:

  1. VAE Loss ($L_{VAE}$): For basic image reconstruction.
  2. GAN Loss ($L_{GAN}$): To ensure photorealism.
  3. ID Loss ($L_{ID}$): Utilizes a pre-trained face recognition network to penalize changes in identity features.
  4. Cycle Consistency Loss ($L_{Cycle}$): Ensures that reversing an attribute change returns the image to its original state.
  5. Mask Loss ($L_{Mask}$): Uses a generated facial mask to penalize changes outside the mask, effectively preserving the background.

Experimental Results

The model was evaluated on the CelebA dataset, focusing on attributes like hair color, age, gender, and skin tone.

Qualitative results of M-AAE: The grid demonstrates continuous attribute manipulation (e.g., age, mustache, glasses) while maintaining identity and background consistency.

Key Insights

  • Sparsity Benefits: Modifying fewer pixels in the latent space avoids "global drift," preventing the entire image from shifting tone when only one attribute is targeted.
  • Background Preservation: Explicitly prioritizing background stability via mask loss is crucial for maintaining the illusion of a realistic edit.
  • Continuous Control: The scalar $\delta$ allows for a spectrum of changes rather than binary transitions.

Quantitative Performance

In user studies, M-AAE consistently outperformed competitors like StarGAN and UNIT. As the complexity of the task increased (e.g., manipulating 3 or 4 attributes simultaneously), M-AAE's performance gap widened. For 3-attribute manipulation, M-AAE achieved an average rank of 1.70, significantly better than StarGAN's 4.24.

Conclusion

M-AAE offers a robust framework for photorealistic face attribute manipulation. While currently dependent on an external mask-generation network, the principles of sparse modification and mask-aware training show great promise for broader image-to-image translation tasks. Future work will explore generalizing these techniques to other domains beyond facial editing.

Summary updated

Abstract

The task of face attribute manipulation has found increasing applications, but still remains challenging with the requirement of editing the attributes of a face image while preserving its unique details. In this paper, we choose to combine the Variational AutoEncoder (VAE) and Generative Adversarial Network (GAN) for photorealistic image generation. We propose an effective method to modify a modest amount of pixels in the feature maps of an encoder, changing the attribute strength continuously without hindering global information. Our training objectives of VAE and GAN are reinforced by the supervision of face recognition loss and cycle consistency loss for faithful preservation of face details. Moreover, we generate facial masks to enforce background consistency, which allows our training to focus on manipulating the foreground face rather than background. Experimental results demonstrate our method, called Mask-Adversarial AutoEncoder (M-AAE), can generate high-quality images with changing attributes and outperforms prior methods in detail preservation.
BibTeX
@article{sun2021mask,
  author = {Ruoqi Sun and Chen Huang and Jianping Shi and Lizhuang Ma},
  title = {{Mask-aware Photorealistic Face Attribute Manipulation}},
  journal = {Comput. Vis. Media},
  year = {2021},
  doi = {10.1007/s41095-021-0219-7},
}
Figure