Decoupling Forgery Semantics for Generalizable Deepfake Detection

Wei Ye, Xinan He, Feng Ding

Abstract

Deepfake detection models frequently struggle with generalization, performing well on known datasets but failing when confronted with novel manipulation techniques. This paper presents a novel framework that improves cross-domain detection by decoupling image features into three distinct categories: common forgery traits, unique forgery artifacts, and irrelevant background content. By isolating the universal "tells" of synthetic manipulation, the proposed method achieves state-of-the-art performance across multiple unseen datasets.

Introduction

The rapid advancement of generative AI has created a "cat-and-mouse" game between deepfake creators and forensic researchers. While current detectors achieve high accuracy in intra-domain scenarios—where the training and testing data share the same generation algorithms—their performance typically plummets in cross-domain evaluations. This lack of generalization is primarily due to models overfitting to "irrelevant content semantics" (such as specific lighting or facial identities) or "unique forgery semantics" (artifacts specific to a single generator).

To address this, we propose a semantic decoupling approach. The goal is to extract "common forgery semantics"—the universal mathematical fingerprints left behind by synthetic manipulation—while discarding the noise that leads to overfitting.

Methodology

The proposed framework utilizes a two-stage training strategy designed to disentangle complex image features into actionable components.

1. Semantic Decoupling Framework

The model employs a dual-encoder architecture, utilizing both SwiftFormer and Xception backbones, to process images through two distinct stages:

  • Stage 1: Content vs. Forgery: The model splits an input image into "irrelevant content" (identity, background) and "all forgery semantics." To ensure the integrity of this separation, a reconstruction task is employed, where a decoder attempts to rebuild the original image from these decoupled features.
  • Stage 2: Common vs. Unique Forgery: The "all forgery" features are further refined. The model disentangles "common forgery semantics" (shared across different manipulation methods) from "unique forgery semantics" (specific to a single algorithm).

During inference, the model discards the unique and irrelevant features, relying exclusively on the common forgery semantics to determine if an image is real or fake.

2. Adaptive High-Pass Filter (AHF)

Standard RGB pixels often mask the subtle anomalies of generative models. To capture these, the framework introduces a learnable Adaptive High-Pass Filter (AHF). Unlike traditional static filters, the AHF adapts during training to identify specific frequency-domain anomalies that human observers and standard CNNs typically miss.

3. Feature Fusion and Contrastive Learning

To integrate these frequency clues with spatial textures, the model uses Multi-scale High-frequency Feature Extraction (MHFE) and Fusion (MHFF) modules. Furthermore, a contrastive loss function is applied to ensure that common forgery features from diverse sources are mapped closely together in the feature space, while unique features remain distinct.

Experimental Results

The model was trained on the FaceForensics++ (FF++) dataset, which includes fakes generated by five different algorithms (DeepFakes, Face2Face, FaceSwap, NeuralTexture, and FaceShifter). Its generalization capabilities were then tested on several unseen datasets.

Cross-Domain Performance

The proposed method consistently outperformed existing state-of-the-art (SOTA) models in cross-domain scenarios:

  • Celeb-DF: On this high-quality dataset with reduced visual artifacts, the model achieved an AUC of 76.94%, surpassing the previous best of 75.19%.
  • DeepfakeDetection (DFD): The model reached 83.02% AUC, a significant improvement over traditional methods that often struggle to exceed the 80% threshold on this diverse dataset.
  • Intra-Domain Accuracy: Despite the focus on generalization, the model maintained near-perfect performance on its training domain, achieving 99.36% on the FaceSwap sub-dataset of FF++.

Discussion and Limitations

While semantic decoupling significantly improves generalization, certain challenges remain. The two-stage training process and the use of multiple encoders/decoders increase computational complexity compared to simpler classification models. Additionally, while the fusion of RGB and frequency data helps, extreme video compression (common on social media platforms) can still degrade the high-frequency signals that the AHF module relies upon.

Conclusion

By shifting the focus from specific artifact detection to the decoupling of universal forgery semantics, this research provides a robust path forward for deepfake forensics. The ability to isolate common traits across different generative techniques ensures that detectors remain effective even as new forgery methods emerge. Future work will focus on further enhancing the independence of these semantics and improving robustness against heavy media compression.

Summary updated

BibTeX
@inproceedings{ye2024decoupling,
  author = {Wei Ye and Xinan He and Feng Ding},
  title = {{Decoupling Forgery Semantics for Generalizable Deepfake Detection}},
  booktitle = {BMVC},
  year = {2024},
}
Figure