Information Bottleneck Disentanglement for Identity Swapping

Gege Gao, Huaibo Huang, Chaoyou Fu, Zhaoyang Li, Ran He

Introduction

Identity swapping, or face swapping, aims to generate a face that inherits the identity of a source image while preserving the perceptual attributes—such as pose, lighting, and expression—of a target image. A persistent challenge in this field is entanglement, where the target's facial geometry (e.g., jawline or eye shape) "leaks" into the final result, creating an ambiguous hybrid identity.

Existing subject-agnostic methods often rely on implicit guidance through multiple loss functions, which frequently fails to achieve clean separation. This paper introduces InfoSwap, a framework that explicitly models disentanglement using the Information Bottleneck (IB) principle to extract "minimal sufficient statistics" for identity, thereby minimizing leakage from the target.

Methodology: The InfoSwap Framework

The InfoSwap architecture is built around a pre-trained face recognition model (ArcFace) and consists of two primary modules designed to enforce explicit disentanglement.

Overview of the InfoSwap architecture, featuring the Informative Identity Bottleneck (IIB) for disentanglement and the Adaptive Information Integrator (AII) for synthesis.

Informative Identity Bottleneck (IIB)

The IIB module treats disentanglement as an information compression task. It employs "information controllers" ($\lambda$) that dynamically add noise to the internal features of the pre-trained backbone. By optimizing for the minimal amount of data required to represent identity, the model learns to discard identity-irrelevant information (like pose and expression) as noise.

Adaptive Information Integrator (AII)

Once the identity is compressed, the AII module integrates this source identity with the target's perceptual features. Utilizing a U-Net decoder and Adaptive Instance Normalization (AdaIN), the AII synthesizes a high-resolution image that blends these disparate components seamlessly.

Identity Contrastive Loss (ICL)

To further refine the identity, the authors introduce the Identity Contrastive Loss. This loss function ensures the generated face is close to the source identity in the latent space while remaining distinctly far from the target identity. This prevents the model from settling for a "middle-ground" face that lies on the angle bisector between the two identities.

Experimental Results

The model was trained on the FFHQ and CelebA-HQ datasets (96,000 images at 512x512 resolution) and evaluated on the FaceForensics++ (FF++) benchmark.

Qualitative Analysis

InfoSwap demonstrates superior preservation of source-specific traits compared to state-of-the-art (SOTA) methods. As shown in the comparison below, the model effectively captures nuances such as face shape, cheekbone structure, and chin length, which are often lost or blended in competing frameworks like FaceShifter or FSGAN.

Qualitative comparison of InfoSwap against other face-swapping methods across various facial attributes.

Quantitative Performance

  • ID Retrieval: InfoSwap achieved a record ID retrieval rate of 99.67% on FF++, significantly outperforming FaceShifter (97.38%) and FSGAN (60.41%).
  • Identity Discriminativeness: Statistical analysis revealed that InfoSwap is the only method where the identity similarity distribution of generated faces has zero overlap with the target identity distribution.
  • User Preference: In a study of 50 users, InfoSwap was rated as "most identical to source" 51.1% of the time, more than double the score of the next best competitor.

Discussion and Future Work

While InfoSwap excels at identity preservation, the authors note that methods like FSGAN may occasionally show lower expression errors by only swapping the inner face. However, InfoSwap’s whole-face approach provides a more realistic and consistent result. Beyond identity swapping, the high-fidelity outputs of InfoSwap could serve as a valuable resource for training more robust forgery detection systems. The researchers acknowledge the potential for misuse and emphasize that their work on discriminative identities can contribute to developing defensive technologies against malicious deepfakes.

Summary updated

Abstract

Improving the performance of face forgery detectors often requires more identity-swapped images of higher-quality. One core objective of identity swapping is to generate identity-discriminative faces that are distinct from the target while identical to the source. To this end, properly disentangling identity and identity-irrelevant information is critical and remains a challenging endeavor. In this work, we propose a novel information disentangling and swapping network, called InfoSwap, to extract the most expressive information for identity representation from a pre-trained face recognition model. The key insight of our method is to formulate the learning of disentangled representations as optimizing an information bottleneck trade-off, in terms of finding an optimal compression of the pre-trained latent features. Moreover, a novel identity contrastive loss is proposed for further disentanglement by requiring a proper distance between the generated identity and the target. While the most prior works have focused on using various loss functions to implicitly guide the learning of representations, we demonstrate that our model can provide explicit supervision for learning disentangled representations, achieving impressive performance in generating more identity-discriminative swapped faces.
BibTeX
@inproceedings{gao2021information,
  author = {Gege Gao and Huaibo Huang and Chaoyou Fu and Zhaoyang Li and Ran He},
  title = {{Information Bottleneck Disentanglement for Identity Swapping}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.00341},
}
Figure