High-Fidelity Face Manipulation With Extreme Poses and Expressions

Chaoyou Fu

TL;DR

This paper introduces a novel two-stage framework for high-fidelity face manipulation, enabling realistic control over extreme poses and expressions even at high resolutions. To address the limitations of existing datasets, they also introduce MVF-HQ, a large-scale, high-resolution multi-view face dataset.

What Problem Does This Paper Address?

Face manipulation, the art of altering facial attributes like pose and expression, has seen significant progress thanks to Generative Adversarial Networks (GANs). However, existing methods struggle with two key challenges: simultaneously controlling both pose and expression, and achieving this at high resolutions, especially for extreme variations (e.g., poses beyond ±60 degrees). Previous approaches often result in blurry structures, lost textures, or require manual landmark annotations. Furthermore, publicly available datasets are often limited in resolution or lack sufficient extreme pose variations, hindering the development and evaluation of robust manipulation techniques.

Methods

The authors propose a two-stage approach:

  1. Boundary Prediction Stage: This stage focuses on jointly modeling poses and expressions by predicting a target boundary image. A conditional encoder-decoder network takes an input face and desired pose/expression vectors to generate the target boundary. To handle unseen poses and expressions, they employ a semi-supervised approach using pre-trained pose and expression estimators. This stage uses a pixel-wise loss for known poses/expressions and a conditional regression loss for unknown ones.
  2. Disentangled Face Synthesis Stage: This stage takes the predicted boundary image and the original input face, encoding them into separate latent spaces for structure and texture. A key innovation here is the use of a proxy network (a pre-trained face recognition network) and a novel feature threshold loss. This loss encourages the learned face features to be close to the proxy's compact features, promoting disentanglement between structure and texture. They also incorporate multi-scale pixel-wise loss, multi-scale conditional adversarial loss, and an identity preserving loss to ensure high-quality, identity-consistent synthesis.

Datasets

  • MultiPIE: Used for training and evaluation, particularly for pose and expression synthesis.
  • RaFD: Used for expression analysis and synthesis.
  • CelebA-HQ: Used for high-resolution face frontalization experiments, especially in uncontrolled "in-the-wild" settings.
  • CelebA: Used to test performance on a dataset with abundant poses but lower image quality.
  • MVF-HQ (Multi-View Face - High Quality): A newly introduced dataset by the authors. It contains 120,283 images at a massive 6000x4000 resolution from 479 identities, covering 13 views from -90° to +90°, diverse expressions, and illuminations. This dataset is significantly larger and higher resolution than existing benchmarks, specifically designed to address the limitations for extreme pose manipulation.

Key Insights and Ideas

  • Boundary as a Joint Representation: Using boundary images as an intermediate representation effectively decouples the complex task of pose and expression control from direct image synthesis.
  • Semi-Supervised Boundary Prediction: The ability to leverage pre-trained estimators for unseen poses/expressions is crucial for generating flexible target boundaries.
  • Feature Threshold Loss for Disentanglement: Directly using features from a face recognition network can lead to texture loss. The proposed feature threshold loss offers a more nuanced way to disentangle structure and texture, preserving both.
  • MVF-HQ Dataset's Importance: The creation of a high-resolution, large-scale dataset with extreme pose variations is critical for advancing research in this challenging area, as existing datasets are insufficient.
  • Two-Stage Approach for High-Fidelity: Separating boundary prediction from synthesis allows for more focused optimization and better control over the final output quality.

Key Results

  • The proposed method achieves high-fidelity face manipulation with clear structures and realistic textures, even for extreme poses (up to ±90°).
  • Qualitative results demonstrate superior performance compared to state-of-the-art methods like CAPG-GAN and pix2pixHD, especially in preserving details and handling large pose changes.
  • Quantitative experiments on MultiPIE show significant improvements in face recognition accuracy when using synthesized frontal faces from extreme poses, particularly at ±90° (achieving 60.4% Rank-1 accuracy with their method vs. 23.9% for pix2pixHD).
  • Experiments on the new MVF-HQ dataset also show superior FID scores and Rank-1 accuracies compared to other methods.
  • The ablation study confirms the necessity of each component of their framework, with the feature threshold loss being particularly important for disentanglement.

Limitations and Future Work

  • Artifacts in Uncontrolled Environments: While the method performs well on controlled datasets, experiments on the "in-the-wild" CelebA dataset show some artifacts, likely due to the low quality of the original images and the challenges of uncontrolled variants.
  • Future Work: The authors plan to continue collecting data to further enrich the MVF-HQ dataset. They also implicitly suggest that further improvements in handling highly uncontrolled environments could be a future direction.

Summary updated

Abstract

Face manipulation has shown remarkable advances with the flourish of Generative Adversarial Networks. However, due to the difficulties of controlling structures and textures, it is challenging to model poses and expressions simultaneously, especially for the extreme manipulation at high-resolution. In this article, we propose a novel framework that simplifies face manipulation into two correlated stages: a boundary prediction stage and a disentangled face synthesis stage. The first stage models poses and expressions jointly via boundary images. Specifically, a conditional encoder-decoder network is employed to predict the boundary image of the target face in a semi-supervised way. Pose and expression estimators are introduced to improve the prediction performance. In the second stage, the predicted boundary image and the input face image are encoded into the structure and the texture latent space by two encoder networks, respectively. A proxy network and a feature threshold loss are further imposed to disentangle the latent space. Furthermore, due to the lack of high-resolution face manipulation databases to verify the effectiveness of our method, we collect a new high-quality Multi-View Face (MVF-HQ) database. It contains 120,283 images at 6000 × 4000 resolution from 479 identities with diverse poses, expressions, and illuminations. MVF-HQ is much larger in scale and much higher in resolution than publicly available high-resolution face manipulation databases. We will release MVF-HQ soon to push forward the advance of face manipulation. Qualitative and quantitative experiments on four databases show that our method dramatically improves the synthesis quality.
BibTeX
@article{fu2021high,
  author = {Chaoyou Fu},
  title = {{High-Fidelity Face Manipulation With Extreme Poses and Expressions}},
  journal = {IEEE Trans. Inf. Forensics Secur.},
  year = {2021},
  url = {https://openreview.net/forum?id=WoPE23l_vPK},
}
Figure