One-shot Face Reenactment

Yunxuan Zhang, Siwei Zhang, Yue He, Cheng Li, Chen Change Loy, Ziwei Liu

Abstract

Face reenactment typically requires extensive video data of a target subject to learn their unique facial traits. This paper introduces a novel one-shot learning framework that transfers the pose and expression of a source video onto a target person using only a single reference image. By disentangling appearance from shape and employing a "FusionNet" to combine neural synthesis with traditional warping, the proposed system achieves high-fidelity reenactment while strictly preserving the target's identity.

Introduction

Face reenactment—the process of animating a target person’s face based on the movements of a source person—traditionally suffers from a massive data bottleneck. Most state-of-the-art methods are "subject-specific," requiring several minutes of target video to capture unique facial nuances. In real-world scenarios, often only a single photo is available, such as for historical figures or social media profiles. Existing one-shot methods frequently suffer from "identity shift," where the reenacted face loses the distinct features of the target person. This research addresses the challenge of performing realistic shape transfer (pose and expression) while strictly preserving identity using only one reference image.

Methodology

The proposed Disentangle-and-Compose Framework separates identity-specific features from motion-related shape information. It consists of three primary components:

  1. Appearance Auto-Encoder ($F$): Extracts identity-specific features like texture, eye color, and lip thickness. To ensure the encoder ignores the expression in the reference photo, a training strategy replaces the target's face contour with a random person's contour, forcing the model to focus on internal features.
  2. Shape Encoder ($E$): A pretrained model that maps the source face into a 15-channel heatmap (parsing maps) representing facial parts (nose, eyes, mouth) and gaze direction.
  3. Semantically Adaptive Decoder ($D$): Acts as the "composer," taking identity features from $F$ and shape maps from $E$. It utilizes SPADE (Spatially-Adaptive Normalization) blocks to inject shape information at multiple scales, ensuring the final image respects the source's pose while maintaining the target's appearance.
The detailed architecture of the One-Shot Face Reenactment framework, highlighting the Shape Encoder (E), Appearance Auto-Encoder (F), and Semantically Adaptive Decoder (D).

Additionally, a FusionNet acts as a refiner. Since neural networks often struggle with fine textures like hair, FusionNet fuses the neural-generated image with a traditionally warped version of the original image using a generated mask to leverage the strengths of both approaches.

Experimental Setup

The model was trained on a subset of CelebA-HQ containing 20,000 high-quality face images. Evaluation was conducted across several datasets to test generalization:

  • CelebA-HQ: Used for unseen identities to test "same-source" performance.
  • FFHQ: A dataset with diverse gender and ethnicity distributions to test "cross-source" generalization.
  • RAF-DB: A real-world, "in-the-wild" dataset to evaluate robustness in unconstrained environments.

Results and Key Insights

The core innovation lies in the explicit separation of identity and shape. If the appearance encoder captures pose information, the result becomes a hybrid of two expressions. By injecting multi-scale feature maps from the appearance encoder into the decoder, the model retains low-level details like skin texture.

Key performance metrics include:

  • Identity Preservation: The model achieved a score of 98.2% on CelebA-HQ, significantly outperforming baselines like PG2 (52.3%) and GauGAN (79.3%).
  • Pose Accuracy: The average head pose error was only 2.72 degrees, indicating high fidelity to source movements.
  • Competitive Performance: The one-shot approach produced results comparable to subject-specific models like ReenactGAN, which require much more data.

Limitations and Future Work

Despite its success, the model faces challenges with complex backgrounds and significant hair movement. Extreme profile views remain difficult because the "hidden" side of the face is not present in the single reference image. Future work will focus on improving temporal consistency to avoid flickering in long video sequences and handling more extreme poses.

Summary updated

BibTeX
@inproceedings{zhang2019one,
  author = {Yunxuan Zhang and Siwei Zhang and Yue He and Cheng Li and Chen Change Loy and Ziwei Liu},
  title = {{One-shot Face Reenactment}},
  booktitle = {BMVC},
  year = {2019},
  url = {https://openreview.net/forum?id=PBMc3mTTYAN},
}
Figure