Abstract
Face swapping aims to transplant a source person’s identity onto a target person’s image while preserving the target's attributes such as expression, pose, and background. Current state-of-the-art (SOTA) models often suffer from a "seesaw" problem, where improving identity preservation leads to the loss of target attributes. This paper identifies skip connections as a primary source of identity leakage and proposes WSC-swap (Without Skip Connection swap). By removing skip connections and utilizing a dual-encoder system for facial and non-facial attributes, the framework achieves superior disentanglement and identity consistency.
Introduction
The goal of face swapping is to create a high-fidelity image that combines a source's identity with a target's context. However, existing architectures, particularly those based on U-Nets, frequently allow the target's facial shape to "leak" into the final output. This phenomenon is largely attributed to skip connections, which, while effective at preserving fine details, inadvertently carry identity-specific information from the target image to the decoder. Furthermore, the conflicting requirements of encoding high-level semantic data (like pose) and low-level pixel data (like background) within a single encoder often lead to suboptimal results.
Methods
The WSC-swap framework addresses these challenges through three core innovations:
- Removal of Skip Connections: By eliminating skip connections, the model prevents the direct transfer of target facial geometry, forcing the decoder to rely on disentangled latent representations.
- Dual Target Encoders: The architecture splits the target encoding task into two specialized components:
- Facial Non-ID (FNID) Encoder: Captures semantic attributes within the face (e.g., expression, pose) while actively stripping away identity using an adversarial head (AdvHead) and 3D Morphable Model (3DMM) priors.
- Non-Facial Attribute (NFA) Encoder: Focuses on pixel-level details outside the facial region, such as hair and background, using a masked-image training strategy.
- Reinforced Disentanglement: The training process incorporates identity removal losses and non-identity preservation losses to ensure the encoders effectively separate identity from other attributes.
Key Results
WSC-swap was evaluated on the FaceForensics++ (FF++) dataset and a novel test split of CelebA-HQ designed to measure identity consistency (ID-Consis).
Quantitative Performance
On the FF++ benchmark, WSC-swap achieved an ID retrieval rate of 99.88%, outperforming previous SOTA models like Faceshifter (97.38%) and Simswap (92.83%). The model also demonstrated significantly lower errors in preserving target poses and expressions.
Qualitative Analysis
Visual evaluations show that WSC-swap excels in handling complex scenarios, including occlusions and extreme poses, where other models often fail. The swapped faces maintain a high degree of consistency across different target backgrounds, avoiding the "identity drifting" common in single-encoder systems.
Conclusion
The WSC-swap framework demonstrates that removing skip connections and employing specialized dual encoders can effectively solve the identity leakage problem in face swapping. While StyleGAN-based methods currently offer higher raw image fidelity, WSC-swap sets a new standard for identity preservation and attribute disentanglement. Future work will explore integrating these disentanglement techniques with high-resolution generative models to achieve both perfect identity transfer and photo-realistic rendering.
Summary updated