High Fidelity Face Manipulation with Extreme Pose and Expression

Chaoyou Fu, Yibo Hu, Xiang Wu, Guoli Wang, Qian Zhang, Ran HearXiv 2019 · arXiv

Abstract

Generating high-resolution facial images under extreme poses and expressions remains a significant challenge for existing generative models. We present a two-stage framework that utilizes dense boundary prediction and a novel Feature Threshold Loss to effectively disentangle facial structure from fine-grained texture. Our approach maintains high fidelity and identity preservation even at 90-degree profile views. Furthermore, we introduce the Multi-View Face High Quality (MVF-HQ) dataset, containing over 120,000 ultra-high-resolution images, to address the scarcity of diverse multi-view data.

Introduction

Face manipulation—the digital alteration of a subject's pose or expression—has seen remarkable progress through Generative Adversarial Networks (GANs). However, current state-of-the-art methods often fail when tasked with high-resolution ($1024 \times 1024$) synthesis involving extreme variations. The primary difficulty lies in the entanglement of facial structure (e.g., the shape of the nose) and texture (e.g., skin pores and hair). Standard disentanglement techniques often result in "blurry" outputs or a loss of the subject's unique identity. This work proposes a framework that overcomes these limitations by providing dense structural guidance and a flexible loss function that preserves high-frequency details.

Methodology

The proposed system decomposes the manipulation task into two correlated stages: boundary prediction and disentangled synthesis.

Boundary Prediction Stage

Rather than using sparse landmarks, which provide insufficient guidance for high-resolution tasks, our model predicts a dense "boundary image." This sketch-like map captures the underlying geometry of the target pose and expression, ensuring structural correctness before pixel synthesis begins.

Boundary maps provide a dense structural signal for high-resolution synthesis.

Disentangled Face Synthesis and Feature Threshold Loss

The synthesis stage employs separate encoders for the predicted boundary (structure) and the source image (texture). To solve the texture-loss problem inherent in identity-preserving networks, we introduce the Feature Threshold Loss. Using a pre-trained face recognition network as a structural proxy, this loss only penalizes differences that exceed a specific margin ($m$). This allows the model to satisfy structural constraints while retaining the fine-grained textures that make a face appear realistic.

The MVF-HQ Dataset

A major bottleneck in training high-fidelity models is the lack of high-resolution datasets with extreme poses. Datasets like CelebA-HQ, while high-quality, are predominantly front-facing. To address this, we introduce MVF-HQ (Multi-View Face High Quality), featuring 120,283 images at $6000 \times 4000$ resolution. The dataset includes 13 camera views ranging from $-90^\circ$ to $+90^\circ$, providing a much-needed benchmark for extreme-angle synthesis.

Comparison of pose distributions across popular datasets highlights the necessity of MVF-HQ for extreme angle manipulation.

Results

Our framework achieves state-of-the-art results on standard benchmarks. On the MultiPIE dataset, the model maintains a 99.9% Rank-1 recognition accuracy for $15^\circ$ poses and 87.4% at $90^\circ$, significantly outperforming previous models like CAPG-GAN (66% at $90^\circ$). Qualitative evaluations at $1024 \times 1024$ resolution show exceptional sharpness in fine details such as teeth and eyes, even during extreme expressions like screaming. Ablation studies confirm that the Feature Threshold Loss is critical for maintaining sharpness, while the identity-preserving loss is essential for recognizing the subject.

Conclusion

By integrating dense boundary guidance with a thresholded disentanglement loss, we enable high-fidelity face manipulation across extreme poses and expressions. The introduction of the MVF-HQ dataset further supports the development of robust, high-resolution multi-view synthesis models. Future research will focus on extending these capabilities to unconstrained, "in-the-wild" environments.

Summary updated

Abstract

Face manipulation has shown remarkable advances with the flourish of Generative Adversarial Networks. However, due to the difficulties of controlling structures and textures, it is challenging to model poses and expressions simultaneously, especially for the extreme manipulation at high-resolution. In this paper, we propose a novel framework that simplifies face manipulation into two correlated stages: a boundary prediction stage and a disentangled face synthesis stage. The first stage models poses and expressions jointly via boundary images. Specifically, a conditional encoder-decoder network is employed to predict the boundary image of the target face in a semi-supervised way. Pose and expression estimators are introduced to improve the prediction performance. In the second stage, the predicted boundary image and the input face image are encoded into the structure and the texture latent space by two encoder networks, respectively. A proxy network and a feature threshold loss are further imposed to disentangle the latent space. Furthermore, due to the lack of high-resolution face manipulation databases to verify the effectiveness of our method, we collect a new high-quality Multi-View Face (MVF-HQ) database. It contains 120,283 images at 6000x4000 resolution from 479 identities with diverse poses, expressions, and illuminations. MVF-HQ is much larger in scale and much higher in resolution than publicly available high-resolution face manipulation databases. We will release MVF-HQ soon to push forward the advance of face manipulation. Qualitative and quantitative experiments on four databases show that our method dramatically improves the synthesis quality.
BibTeX
@misc{fu2019high,
  author = {Chaoyou Fu and Yibo Hu and Xiang Wu and Guoli Wang and Qian Zhang and Ran He},
  title = {{High Fidelity Face Manipulation with Extreme Pose and Expression}},
  year = {2019},
  eprint = {1903.12003},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1903.12003},
}
Figure