Abstract
Face Forgery Detection (FFD), or Deepfake detection, aims to determine whether a digital face is real or fake. Due to different face synthesis algorithms with diverse forgery patterns, FFD models often overfit specific patterns in training datasets, resulting in poor generalization to other unseen forgeries. Existing FFD methods primarily leverage pre-trained backbones with general image representation capabilities and fine-tune them to identify facial forgery cues. However, these backbones lack domain-specific facial knowledge and insufficiently capture complex facial features, thus hindering effective implicit forgery cue identification and limiting generalization. Therefore, it is essential to revisit FFD workflow across the pre-training and fine-tuning stages, achieving an elaborate integration from facial representation to forgery detection to improve generalization. Specifically, we develop an FFD-specific pre-trained backbone with superior facial representation capabilities through self-supervised pre-training on real faces. We then propose a competitive fine-tuning framework that stimulates the backbone to identify implicit forgery cues through a competitive learning mechanism. Moreover, we devise a threshold optimization mechanism that utilizes prediction confidence to improve the inference reliability. Comprehensive experiments demonstrate that our method achieves excellent performance in FFD and extra face-related tasks, i.e., presentation attack detection.
Introduction
Face Forgery Detection (FFD) has become a critical area of research as synthesis techniques grow increasingly sophisticated. A persistent challenge in the field is the "generalization gap," where models performing exceptionally well on training data fail when exposed to unseen forgery algorithms. This is largely due to the reliance on backbones pre-trained on general object datasets like ImageNet. Such models are optimized for broad object recognition rather than the subtle, local inconsistencies—such as irregular textures or component distortions—that signal a facial forgery.
Methodology: The Revitalized FFD Pipeline
We propose a revitalized FFD pipeline that intervenes at three critical stages: pre-training, fine-tuning, and inference. This approach shifts the focus from general image features to domain-specific facial representations.
1. Face-Specific Pre-training
Instead of supervised learning on general images, we utilize Self-Supervised Learning (SSL) on a massive dataset of real faces. We found that Masked Image Modeling (MIM), specifically using the BEiT v2 framework, is exceptionally effective. By forcing the model to reconstruct missing facial patches, it learns to represent local facial components (eyes, nose, mouth) with high precision—exactly where forgery cues often hide.
2. Competitive Backbone Fine-tuning
To prevent the model from overfitting to easy-to-spot artifacts, we use a dual-branch framework. Two unshared backbones process the same image, constrained by a decorrelation loss ($L_{dec}$) to ensure they learn distinct features. An Uncertainty-based Fusion Module (UFM) dynamically weights these branches, giving more influence to the more confident one. This competition forces the branches to uncover more robust, implicit forgery cues.
3. Uncertainty-based Threshold Optimization (UTOM)
Standard models often use a fixed 0.5 probability threshold, which may not be optimal for unseen datasets. Our UTOM mechanism utilizes prediction probability and uncertainty to calculate a more reliable, adaptive threshold for inference, enhancing the model's robustness in real-world scenarios.
Experimental Results
We evaluated our approach using FaceForensics++ (FF++) for training and several major datasets (Celeb-DF, DFDC, FFIW) for cross-dataset evaluation.
As shown in the results, the transition from general ImageNet pre-training to face-specific SSL pre-training yields striking improvements. For example, using BEiT v2 pre-trained on faces with a ViT-B backbone jumps the average cross-dataset AUC from approximately 61% to 86.56%. Our final model, CFB++, achieves a state-of-the-art average AUC of 93.35% across five major benchmarks, significantly outperforming existing methods.
Conclusion
By revisiting FFD through the lens of facial representation, we demonstrate that domain-specific pre-training combined with competitive learning mechanisms can effectively bridge the generalization gap. Our revitalized pipeline provides a more robust and reliable foundation for detecting increasingly realistic digital forgeries.
Summary updated