Abstract
Existing face forgery detection methods usually treat face forgery detection as a binary classification problem and adopt deep convolution neural networks to learn discriminative features. The ideal discriminative features should be only related to the real/fake labels of facial images. However, we observe that the features learned by vanilla classification networks are correlated to unnecessary properties, such as forgery methods and facial identities. Such phenomenon would limit forgery detection performance especially for the generalization ability. Motivated by this, we propose a novel method which utilizes adversarial learning to eliminate the negative effect of different forgery methods and facial identities, which helps classification network to learn intrinsic common discriminative features for face forgery detection. To leverage data lacking ground truth label of facial identities, we design a special identity discriminator based on similarity information derived from off-the-shelf face recognition model. With the help of adversarial learning, our face forgery detection model learns to extract common discriminative features through eliminating the effect of forgery methods and facial identities. Extensive experiments demonstrate the effectiveness of the proposed method under both intra-dataset and cross-dataset evaluation settings.
Introduction
As deepfake technology continues to evolve, the ability to create highly realistic facial manipulations has outpaced the development of robust detection tools. Most current face forgery detection methods rely on deep convolutional neural networks (CNNs) trained as binary classifiers. While these models perform exceptionally well on datasets they have already seen, they often struggle in real-world scenarios where they encounter unfamiliar faces or novel manipulation techniques.
Researchers from USTC identified a core reason for this failure: standard classification networks inadvertently learn "shortcut" features. Instead of identifying the fundamental essence of a forgery, these models often rely on features specific to a particular forgery method (e.g., specific artifacts from FaceSwap) or even the facial identity of the person in the video. This paper proposes a multi-task adversarial learning framework designed to "strip away" these irrelevant features, forcing the model to focus on intrinsic, common discriminative features that define a forgery regardless of the context.
Methodology
The proposed framework aims to decouple the forgery detection task from the confounding variables of identity and manipulation technique. It consists of three primary components integrated into a unified training pipeline:
1. Feature Generator and Binary Classifier
This is the core of the detection system. The Feature Generator extracts high-level representations from input facial images, which the Binary Classifier then uses to determine if the image is real or fake.
2. Adversarial Discriminators
To ensure the Feature Generator ignores irrelevant information, the authors introduce two adversarial components:
- Forgery Method Discriminator: This component attempts to identify which specific manipulation tool (e.g., Deepfakes, Face2Face, FaceSwap, or NeuralTextures) was used to create the image.
- Identity Discriminator: This component attempts to distinguish between different individuals in the dataset.
3. The Gradient Reversal Layer (GRL)
The "adversarial" nature of the training is facilitated by the Gradient Reversal Layer. During the forward pass, the GRL acts as an identity transform. However, during backpropagation, it multiplies the gradients by a negative constant. This means that while the discriminators try to minimize their error (getting better at identifying methods and people), the Feature Generator is updated to maximize that error. Consequently, the generator learns to produce features that are "blind" to identity and forgery methods, retaining only the information necessary for binary classification.
Handling Unlabeled Identity Data
A significant challenge in this approach is that many deepfake datasets do not provide ground-truth identity labels. To overcome this, the researchers developed an Identity Similarity Supervision module. By using a pre-trained face recognition model (ArcFace), the system calculates similarity scores between different faces. These scores serve as a proxy for identity labels, allowing the identity discriminator to function even in the absence of explicit metadata.
Experimental Results
The model was evaluated using the FaceForensics++ (FF++), Celeb-DF (v2), and DeepFakeDetection (DFD) datasets. The results highlight two major strengths of the adversarial approach:
Cross-Dataset Generalization
One of the most rigorous tests for a deepfake detector is training it on one dataset and testing it on another. When trained on FF++ and tested on the high-quality Celeb-DF dataset, the proposed method achieved an AUC of 79.69%. This significantly outperforms the standard Xception baseline (68.48%), proving that the learned "intrinsic" features are much more transferable to unseen forgery styles.
Robustness to Compression
Digital forensics is often hindered by video compression, which destroys fine-grained textures. In tests on the "Low Quality" (LQ) version of FF++, the proposed model achieved a 94.74% AUC. This suggests that by ignoring identity and method-specific noise, the model captures more robust structural features that survive heavy compression better than traditional methods.
Feature Visualization
Using t-SNE visualizations, the researchers demonstrated that in vanilla models, features for "real" and "fake" images are often clustered by the identity of the subject. In the proposed model, these clusters disappear, and the features are cleanly separated into two distinct groups: real and fake. This confirms that the adversarial training successfully removed identity-related "distractions" from the feature space.
Conclusion
By shifting the focus from simple binary classification to the learning of intrinsic, common discriminative features, this research provides a more generalized solution to the problem of face forgery detection. The use of adversarial learning to eliminate the influence of facial identity and manipulation techniques allows the model to perform reliably across different datasets and video qualities.
Future work may involve extending this frame-based approach to include temporal analysis. While the current model is highly effective at analyzing individual images, incorporating the inconsistencies found between consecutive video frames (such as flickering) could further enhance detection accuracy and robustness against sophisticated temporal forgeries.
Summary updated