Reinforced Disentanglement for Face Swapping without Skip Connection

Xiaohang Ren, Xingyu Chen, Pengfei Yao, Heung-Yeung Shum, Baoyuan Wang

Abstract

Face swapping aims to transplant a source person’s identity onto a target person’s image while preserving the target's attributes such as expression, pose, and background. Current state-of-the-art (SOTA) models often suffer from a "seesaw" problem, where improving identity preservation leads to the loss of target attributes. This paper identifies skip connections as a primary source of identity leakage and proposes WSC-swap (Without Skip Connection swap). By removing skip connections and utilizing a dual-encoder system for facial and non-facial attributes, the framework achieves superior disentanglement and identity consistency.

Introduction

The goal of face swapping is to create a high-fidelity image that combines a source's identity with a target's context. However, existing architectures, particularly those based on U-Nets, frequently allow the target's facial shape to "leak" into the final output. This phenomenon is largely attributed to skip connections, which, while effective at preserving fine details, inadvertently carry identity-specific information from the target image to the decoder. Furthermore, the conflicting requirements of encoding high-level semantic data (like pose) and low-level pixel data (like background) within a single encoder often lead to suboptimal results.

Methods

The WSC-swap framework addresses these challenges through three core innovations:

  1. Removal of Skip Connections: By eliminating skip connections, the model prevents the direct transfer of target facial geometry, forcing the decoder to rely on disentangled latent representations.
  2. Dual Target Encoders: The architecture splits the target encoding task into two specialized components:
  3. Facial Non-ID (FNID) Encoder: Captures semantic attributes within the face (e.g., expression, pose) while actively stripping away identity using an adversarial head (AdvHead) and 3D Morphable Model (3DMM) priors.
  4. Non-Facial Attribute (NFA) Encoder: Focuses on pixel-level details outside the facial region, such as hair and background, using a masked-image training strategy.
  5. Reinforced Disentanglement: The training process incorporates identity removal losses and non-identity preservation losses to ensure the encoders effectively separate identity from other attributes.
Comparison of standard face-swapping architectures (a) with skip connections versus the proposed WSC-swap (b) which removes them and uses dual encoders.

Key Results

WSC-swap was evaluated on the FaceForensics++ (FF++) dataset and a novel test split of CelebA-HQ designed to measure identity consistency (ID-Consis).

Quantitative Performance

On the FF++ benchmark, WSC-swap achieved an ID retrieval rate of 99.88%, outperforming previous SOTA models like Faceshifter (97.38%) and Simswap (92.83%). The model also demonstrated significantly lower errors in preserving target poses and expressions.

Qualitative Analysis

Visual evaluations show that WSC-swap excels in handling complex scenarios, including occlusions and extreme poses, where other models often fail. The swapped faces maintain a high degree of consistency across different target backgrounds, avoiding the "identity drifting" common in single-encoder systems.

Qualitative results of WSC-swap showing the successful transfer of source identity onto various target attributes and poses.

Conclusion

The WSC-swap framework demonstrates that removing skip connections and employing specialized dual encoders can effectively solve the identity leakage problem in face swapping. While StyleGAN-based methods currently offer higher raw image fidelity, WSC-swap sets a new standard for identity preservation and attribute disentanglement. Future work will explore integrating these disentanglement techniques with high-resolution generative models to achieve both perfect identity transfer and photo-realistic rendering.

Summary updated

Abstract

The SOTA face swap models still suffer the problem of either target identity (i.e., shape) being leaked or the target non-identity attributes (i.e., background, hair) failing to be fully preserved in the final results. We show that this insufficient disentanglement is caused by two flawed designs that were commonly adopted in prior models: (1) counting on only one compressed encoder to represent both the semantic-level non-identity facial attributes(i.e., pose) and the pixel-level non-facial region details, which is contradictory to satisfy at the same time; (2) highly relying on long skip-connections between the encoder and the final generator, leaking a certain amount of target face identity into the result. To fix them, we introduce a new face swap framework called 'WSC-swap' that gets rid of skip connections and uses two target encoders to respectively capture the pixel-level non-facial region attributes and the semantic non-identity attributes in the face region. To further reinforce the disentanglement learning for the target encoder, we employ both identity removal loss via adversarial training (i.e., GAN) and the non-identity preservation loss via prior 3DMM models like [11]. Extensive experiments on both FaceForensics++ and CelebA-HQ show that our results significantly outperform previous works on a rich set of metrics, including one novel metric for measuring identity consistency that was completely neglected before.
BibTeX
@inproceedings{ren2023reinforced,
  author = {Xiaohang Ren and Xingyu Chen and Pengfei Yao and Heung-Yeung Shum and Baoyuan Wang},
  title = {{Reinforced Disentanglement for Face Swapping without Skip Connection}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure