FaceDancer: Pose- and Occlusion-Aware High Fidelity Face Swapping

Felix Rosberg, Eren Erdal Aksoy, Fernando Alonso-Fernandez, Cristofer EnglundWACV 2023

TL;DR

FaceDancer is a novel single-stage method for high-fidelity face swapping that excels at preserving target attributes like pose and expression while accurately transferring source identity. It introduces two key innovations: Adaptive Feature Fusion Attention (AFFA) for adaptive feature integration and Interpreted Feature Similarity Regularization (IFSR) for robust attribute preservation, outperforming existing methods on challenging datasets.

What Problem Does This Paper Address?

Face swapping, the task of replacing a person's face in an image or video with another's while maintaining the original scene's attributes, has numerous applications in film, gaming, and entertainment. However, achieving high-fidelity results remains challenging. Existing methods often struggle to simultaneously preserve crucial target attributes like head pose, facial expression, and lighting, while accurately transferring the source identity. Furthermore, many methods require complex multi-stage pipelines or additional processing steps like facial segmentation, which can be computationally expensive and prone to errors. This paper aims to address these limitations by proposing a unified, efficient, and robust single-stage approach.

Methods

FaceDancer employs a conditional Generative Adversarial Network (GAN) framework with a generator and a discriminator, incorporating a mapping network and the ArcFace identity encoder.

  • Adaptive Feature Fusion Attention (AFFA): This novel module is embedded within the decoder. It adaptively learns to fuse attribute features from the target face with identity-conditioned features from the source. Crucially, AFFA does not require explicit facial segmentation. Instead, it learns attention masks to intelligently decide which features to keep and which to discard, thereby improving identity transfer.
  • Interpreted Feature Similarity Regularization (IFSR): This technique enhances attribute preservation by leveraging intermediate features from a pre-trained identity encoder (ArcFace). IFSR explores the similarity between features of the target, source, and generated faces to encourage the preservation of facial expression, head pose, and lighting. It defines margins based on feature similarity distributions to regularize the generator.
  • Generator Architecture: A U-Net-like encoder-decoder structure is used, with residual blocks that employ Adaptive Instance Normalization (AdaIN) or AFFA modules. Skip connections are utilized to pass information from the encoder to the decoder.
  • Mapping Network: A separate mapping network transforms the initial identity distribution to better inject identity information into the generator.
  • Loss Functions: The training objective includes adversarial loss, identity loss, reconstruction loss, perceptual loss, cycle consistency loss, and the proposed IFSR loss.

Datasets

The paper uses the following datasets:

  • VGGFace2: Used for training the FaceDancer model.
  • LS3D-W: Also used for training.
  • FaceForensics++: Used for quantitative evaluation of face swapping performance.
  • AFLW2000-3D: Used for evaluating performance on pose-challenging scenarios.

Key Insights and Ideas

  • Adaptive Feature Gating is Crucial: The AFFA module demonstrates that intelligently fusing features from different sources (target attributes vs. source identity) is key to achieving both good identity transfer and attribute preservation, without needing explicit segmentation.
  • Intermediate Features Hold Attribute Secrets: The IFSR method highlights that specific layers within identity encoders capture distinct types of attribute information (e.g., early layers for pose/expression, later layers for identity). This insight allows for targeted regularization to preserve these attributes.
  • Single-Stage Efficiency: By integrating these novel components into a single-stage framework, FaceDancer achieves competitive results while being more efficient than multi-stage approaches.
  • Robustness to Low-Resolution and Distortions: The proposed method shows remarkable resilience to low-resolution images and artifacts, outperforming other methods in maintaining facial structure and identity under such conditions.

Key Results

  • Quantitative Performance: FaceDancer achieves state-of-the-art identity retrieval (ID) performance on the FaceForensics++ dataset, outperforming methods like SimSwap, FaceShifter, and HifiFace. It also demonstrates competitive pose preservation, achieving the second-lowest pose error among evaluated methods.
  • Qualitative Performance: Visual comparisons show that FaceDancer produces substantially improved identity transfer compared to SimSwap. It handles attributes like facial hair, occlusion, and lighting effectively. On low-resolution images, FaceDancer successfully captures semantic structure and avoids artifacts that plague other methods like SimSwap.
  • Ablation Studies: The studies confirm the significant contributions of both AFFA and IFSR. Removing IFSR leads to a "pasted face" effect and poor expression preservation. Different configurations of AFFA demonstrate its impact on FID scores and color consistency.

Limitations and Future Work

  • Face Shape Transfer: The authors acknowledge that transferring the exact face shape of the identity can be a limitation.
  • IFSR Margin Calculation: The current IFSR method requires calculating margins from a pre-trained face swap model. Future work aims to develop adaptive, online methods for calculating these margins.
  • Future Directions: The paper suggests exploring the combination of IFSR with 3D Morphable Models (3DMM) for stronger pose and face shape preservation. Additionally, IFSR could be used to compress complex face swap models.

Summary updated

BibTeX
@inproceedings{rosberg2023facedancer,
  author = {Felix Rosberg and Eren Erdal Aksoy and Fernando Alonso-Fernandez and Cristofer Englund},
  title = {{FaceDancer: Pose- and Occlusion-Aware High Fidelity Face Swapping}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure