Introduction
The proliferation of deepfake technology has necessitated the development of robust forgery detection systems. However, most current detection models are heavily dependent on the quality of their training data. A significant bottleneck in existing identity-swapping methods is the "appearance gap"—the discrepancy in skin tone and lighting between source and target faces. When these factors are poorly matched, the resulting forgeries contain obvious artifacts that are easily detected by simple models but fail to prepare detectors for sophisticated, real-world "hard" examples. To bridge this gap, we present Appearance Optimal Transport (AOT), a novel framework that leverages optimal transport theory to generate highly realistic face swaps.
Methodology: Appearance Optimal Transport
The AOT framework is designed to treat identity swapping as an optimal transport problem, ensuring that the appearance of the source identity is seamlessly integrated into the target environment.
Relighting Generator (RG)
The core of the AOT framework is the Relighting Generator. Unlike traditional color transfer methods that operate on simple pixel statistics, AOT utilizes Neural Optimal Transport Plan Estimation (NOTPE). This approach solves the optimal transport problem within a high-dimensional latent space, ensuring smoother transitions and more realistic skin textures. To maintain physical plausibility, the model incorporates 3D geometry information, such as surface normals and coordinates derived from a 3D Morphable Model (3DMM). This allows the generator to understand how light interacts with the specific contours of the face, preventing the "flat" look common in lesser swapping methods.
Mix-and-segment Discriminator (MSD)
To further refine the output at the pixel level, we introduce a Mix-and-segment Discriminator. This component frames the training as a "segmentation game." By creating mixed images containing patches of both real and synthetic faces, the discriminator is tasked with segmenting these regions. This forces the generator to minimize the Wasserstein distance—a measure of the difference between two probability distributions—between the real and fake distributions. This process effectively eliminates local inconsistencies in skin texture and tone that traditional GAN discriminators might overlook.
Experimental Results
The AOT framework was evaluated on the FaceForensics++ (FF++) and DeeperForensics-1.0 (DPF-1.0) benchmarks. DPF-1.0 is particularly challenging due to its "in-the-wild" lighting conditions, making it an ideal testbed for appearance matching.
Performance Metrics
AOT consistently outperformed industry standards such as DeepFaceLab and FSGAN. Quantitatively, the method achieved superior scores in Structural Similarity (SSIM) and Gram loss, indicating high fidelity in both structure and texture. A user study involving 31 participants further confirmed these findings; AOT-generated videos were perceived as more realistic and better at preserving target appearance than competing methods significantly more often.
Impact on Forgery Detection
The primary goal of AOT is to improve the robustness of forgery detectors. By augmenting the training sets of standard detectors (e.g., I3D and TSN) with just 100 AOT-generated videos, we observed a significant increase in detection accuracy on unseen test sets. This demonstrates that providing "harder" examples during training directly translates to better security performance in real-world scenarios where lighting is complex.
Conclusion
AOT represents a significant advancement in realistic identity swapping by leveraging optimal transport theory and 3D geometric priors. While the method currently relies on an underlying swapping backbone and can be sensitive to heavy occlusions, it provides a powerful tool for generating high-quality training data. Future research will focus on integrating appearance transport directly into the initial swapping process and improving the handling of occluded facial regions via semantic parsing.
Summary updated