AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection

Hao Zhu, Chaoyou Fu, Qianyi Wu, Wayne Wu, Chen Qian, Ran He

Introduction

Recent studies have shown that the performance of face forgery detection systems heavily relies on the diversity and difficulty of the Deepfakes datasets used for training. However, most identity swapping methods struggle when faced with large "appearance gaps"—such as massive discrepancies in skin tone and lighting between the source and target faces.

To address this, researchers have introduced AOT (Appearance Optimal Transport), a novel framework that uses Optimal Transport theory to bridge these gaps. By generating more realistic and challenging "hard examples," AOT significantly improves the robustness and accuracy of face forgery detection systems.

The Challenge of Appearance Gaps

Most identity swapping technologies work reasonably well when the source and target faces share similar lighting conditions and skin colors. However, they often fall apart when translating a face from a brightly lit outdoor scene onto a target in a dimly lit room, or when swapping faces with vastly different skin tones. These failures result in obvious visual artifacts, "ghosting," or inconsistent coloring.

Visual comparison of identity swapping under extreme lighting and color differences. AOT (Ours) produces significantly fewer artifacts than DeepFaceLab and Conventional OT.

For the research community, this is a critical data problem. If forgery detection algorithms are only trained on "easy" forgeries with minimal appearance gaps, they will fail to detect more sophisticated or varied real-world forgeries. AOT aims to handle these extreme appearance translations to build better datasets for training robust detectors.

Methodology: Optimal Transport in Latent and Pixel Space

The core innovation of AOT is formulating appearance mapping as an Optimal Transport (OT) problem. In simple terms, OT finds the most efficient way to transform one probability distribution (the source appearance) into another (the target appearance). Solving OT directly on pixel histograms often leads to discontinuities and mismatched colors. By moving the OT problem into a learned latent space, the model achieves much smoother and more natural transitions.

The AOT framework consists of two main stages:

  1. Neural Optimal Transport Plan Estimation (NOTPE): Instead of using slow, traditional optimization to solve the OT problem, AOT uses a neural network to estimate the transport plan in a high-dimensional latent space. It employs a two-branch perceptual encoder that incorporates not just image features, but also 3D geometry (coordinates and surface normals). Because lighting interacts with the 3D structure of the face, feeding 3D normals and PNCC (Position Normal Color Code) into the encoder ensures that highlights and shadows land exactly where they naturally should.
  2. Mix-and-segment Discriminator (MSD): To refine the results in the pixel space, the authors introduce a "segmentation game." They create a mixed image containing both real and fake patches. A discriminator is tasked with segmenting the fake parts from the real ones. This fine-grained, patch-based supervision forces the Relighting Generator to produce results so realistic that the discriminator can no longer distinguish between the original target pixels and the newly generated ones.
The AOT architecture featuring the Relighting Generator with NOTPE for latent space transport, and the Mix-and-segment Discriminator for pixel-space refinement.

Key Results and Impact

The authors evaluated their method on two major benchmarks: FaceForensics++ (FF++) and DeeperForensics-1.0 (DPF-1.0), the latter being specifically collected under challenging and diverse lighting conditions.

  • Quantitative Superiority: In comparisons against state-of-the-art methods like DeepFaceLab and FSGAN, AOT achieved the best scores in structural similarity (SSIM-whole) and style/color accuracy (Gram loss).
  • User Preference: In a user study with 31 volunteers, AOT-generated images were picked as the most realistic and best at preserving target appearance significantly more often than competing methods. On DPF-1.0, AOT's realism was preferred roughly 55% of the time, compared to just 15-19% for baselines.
  • Boosting Detection: The ultimate goal of AOT is to improve forgery detection. By adding just 100 AOT-generated videos to the training sets of standard detectors (like I3D and TSN), the authors observed a measurable increase in detection accuracy on hidden test sets.

Limitations and Future Work

The primary limitation of AOT is its handling of occlusions. Because AOT acts as a refinement stage for existing identity-swapping "backbones," it cannot easily recover lost details if the initial backbone fails to handle an occlusion (such as a hand in front of the face or thick glasses frames). Future work may involve integrating the AOT logic more deeply into the initial swapping process or providing explicit parsing maps to handle complex occlusions more gracefully.

Summary updated

Abstract

Recent studies have shown that the performance of forgery detection can be improved with diverse and challenging Deepfakes datasets. However, due to the lack of Deepfakes datasets with large variance in appearance, which can be hardly produced by recent identity swapping methods, the detection algorithm may fail in this situation. In this work, we provide a new identity swapping algorithm with large differences in appearance for face forgery detection. The appearance gaps mainly arise from the large discrepancies in illuminations and skin colors that widely exist in real-world scenarios. However, due to the difficulties of modeling the complex appearance mapping, it is challenging to transfer fine-grained appearances adaptively while preserving identity traits. This paper formulates appearance mapping as an optimal transport problem and proposes an Appearance Optimal Transport model (AOT) to formulate it in both latent and pixel space. Specifically, a relighting generator is designed to simulate the optimal transport plan. It is solved via minimizing the Wasserstein distance of the learned features in the latent space, enabling better performance and less computation than conventional optimization. To further refine the solution of the optimal transport plan, we develop a segmentation game to minimize the Wasserstein distance in the pixel space. A discriminator is introduced to distinguish the fake parts from a mix of real and fake image patches. Extensive experiments reveal that the superiority of our method when compared with state-of-the-art methods and the ability of our generated data to improve the performance of face forgery detection.
BibTeX
@inproceedings{zhu2020aot,
  author = {Hao Zhu and Chaoyou Fu and Qianyi Wu and Wayne Wu and Chen Qian and Ran He},
  title = {{AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=1489ABOTTP},
}
Figure