AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection

Chaoyou Fu

Introduction

The proliferation of deepfake technology has necessitated the development of robust forgery detection systems. However, most current detection models are heavily dependent on the quality of their training data. A significant bottleneck in existing identity-swapping methods is the "appearance gap"—the discrepancy in skin tone and lighting between source and target faces. When these factors are poorly matched, the resulting forgeries contain obvious artifacts that are easily detected by simple models but fail to prepare detectors for sophisticated, real-world "hard" examples. To bridge this gap, we present Appearance Optimal Transport (AOT), a novel framework that leverages optimal transport theory to generate highly realistic face swaps.

Methodology: Appearance Optimal Transport

The AOT framework is designed to treat identity swapping as an optimal transport problem, ensuring that the appearance of the source identity is seamlessly integrated into the target environment.

The AOT framework architecture, illustrating the Relighting Generator (RG) and the Mix-and-segment Discriminator (MSD).

Relighting Generator (RG)

The core of the AOT framework is the Relighting Generator. Unlike traditional color transfer methods that operate on simple pixel statistics, AOT utilizes Neural Optimal Transport Plan Estimation (NOTPE). This approach solves the optimal transport problem within a high-dimensional latent space, ensuring smoother transitions and more realistic skin textures. To maintain physical plausibility, the model incorporates 3D geometry information, such as surface normals and coordinates derived from a 3D Morphable Model (3DMM). This allows the generator to understand how light interacts with the specific contours of the face, preventing the "flat" look common in lesser swapping methods.

Mix-and-segment Discriminator (MSD)

To further refine the output at the pixel level, we introduce a Mix-and-segment Discriminator. This component frames the training as a "segmentation game." By creating mixed images containing patches of both real and synthetic faces, the discriminator is tasked with segmenting these regions. This forces the generator to minimize the Wasserstein distance—a measure of the difference between two probability distributions—between the real and fake distributions. This process effectively eliminates local inconsistencies in skin texture and tone that traditional GAN discriminators might overlook.

Experimental Results

The AOT framework was evaluated on the FaceForensics++ (FF++) and DeeperForensics-1.0 (DPF-1.0) benchmarks. DPF-1.0 is particularly challenging due to its "in-the-wild" lighting conditions, making it an ideal testbed for appearance matching.

Qualitative comparison showing AOT's superior performance in matching skin tone and lighting compared to DeepFaceLab and conventional OT methods.

Performance Metrics

AOT consistently outperformed industry standards such as DeepFaceLab and FSGAN. Quantitatively, the method achieved superior scores in Structural Similarity (SSIM) and Gram loss, indicating high fidelity in both structure and texture. A user study involving 31 participants further confirmed these findings; AOT-generated videos were perceived as more realistic and better at preserving target appearance than competing methods significantly more often.

Impact on Forgery Detection

The primary goal of AOT is to improve the robustness of forgery detectors. By augmenting the training sets of standard detectors (e.g., I3D and TSN) with just 100 AOT-generated videos, we observed a significant increase in detection accuracy on unseen test sets. This demonstrates that providing "harder" examples during training directly translates to better security performance in real-world scenarios where lighting is complex.

Conclusion

AOT represents a significant advancement in realistic identity swapping by leveraging optimal transport theory and 3D geometric priors. While the method currently relies on an underlying swapping backbone and can be sensitive to heavy occlusions, it provides a powerful tool for generating high-quality training data. Future research will focus on integrating appearance transport directly into the initial swapping process and improving the handling of occluded facial regions via semantic parsing.

Summary updated

Abstract

Recent studies have shown that the performance of forgery detection can be improved with diverse and challenging Deepfakes datasets. However, due to the lack of Deepfakes datasets with large variance in appearance, which can be hardly produced by recent identity swapping methods, the detection algorithm may fail in this situation. In this work, we provide a new identity swapping algorithm with large differences in appearance for face forgery detection. The appearance gaps mainly arise from the large discrepancies in illuminations and skin colors that widely exist in real-world scenarios. However, due to the difficulties of modeling the complex appearance mapping, it is challenging to transfer fine-grained appearances adaptively while preserving identity traits. This paper formulates appearance mapping as an optimal transport problem and proposes an Appearance Optimal Transport model (AOT) to formulate it in both latent and pixel space. Specifically, a relighting generator is designed to simulate the optimal transport plan. It is solved via minimizing the Wasserstein distance of the learned features in the latent space, enabling better performance and less computation than conventional optimization. To further refine the solution of the optimal transport plan, we develop a segmentation game to minimize the Wasserstein distance in the pixel space. A discriminator is introduced to distinguish the fake parts from a mix of real and fake image patches. Extensive experiments reveal that the superiority of our method when compared with state-of-the-art methods and the ability of our generated data to improve the performance of face forgery detection.
BibTeX
@inproceedings{fu2020aot,
  author = {Chaoyou Fu},
  title = {{AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=1489ABOTTP},
}
Figure