Abstract
Although vanilla Convolutional Neural Network (CNN) based detectors can achieve satisfactory performance on fake face detection, we observe that the detectors tend to seek forgeries on a limited region of face, which reveals that the detectors is short of understanding of forgery. Therefore, we propose an attention-based data augmentation framework to guide detector refine and enlarge its attention. Specifically, our method tracks and occludes the Top-N sensitive facial regions, encouraging the detector to mine deeper into the regions ignored before for more representative forgery. Especially, our method is simple-to-use and can be easily integrated with various CNN models. Extensive experiments show that the detector trained with our method is capable to separately point out the representative forgery of fake faces generated by different manipulation techniques, and our method enables a vanilla CNN-based detector to achieve state-of-the-art performance without structure modification.
Introduction
Deepfake detectors often "cheat" by overfitting to a few obvious artifacts in limited facial regions, making them brittle against new manipulation techniques. While Convolutional Neural Networks (CNNs) are highly effective at spotting fake faces, they suffer from a "narrow-mindedness" problem. Vanilla detectors tend to focus on very specific, limited regions—like a jagged edge around the chin or a blur near the eyes—to make their decision.
This is problematic for two reasons: first, if a new deepfake method fixes those specific artifacts, the detector fails; second, the model isn't learning what a "forgery" looks like globally. Humans usually scan the entire face for inconsistencies, whereas AI often behaves like a lazy detective who stops looking once they find a single fingerprint. The challenge is to force the AI to look at the "whole crime scene."
Methodology: Representative Forgery Mining (RFM)
The authors propose a framework called Representative Forgery Mining (RFM), which consists of two main components that run during the training loop:
- Forgery Attention Map (FAM): To know where the detector is looking, FAM uses the gradients of the input image relative to the model's output. By calculating the difference in sensitivity between the "real" and "fake" logits, FAM creates a high-resolution, pixel-level map of exactly which facial regions the model finds most suspicious.
- Suspicious Forgeries Erasing (SFE): Once the "Top-N" most sensitive regions are identified via FAM, the SFE module occludes them with random noise.
By hiding the model's "favorite" clues, the training process forces the backbone network to search for other representative forgeries in regions it previously ignored. This iterative process of "hide and seek" results in a detector that understands forgery features across the entire face.
Experiments and Evaluation
The researchers evaluated RFM on two major benchmarks: DFFD (Diverse Fake Face Dataset) and Celeb-DF. DFFD contains both "one-stage" manipulations (like StyleGAN/PGGAN) and "two-stage" manipulations (like FaceSwap/Deepfakes).
Additionally, the authors created a custom "Less-Forgery" test set. They used facial landmarks to segment faces into parts (eyes, nose, mouth, skin) and replaced fake regions with real pixels to see if the model could still detect the forgery using only the remaining, more subtle clues.
Key Results and Insights
- State-of-the-Art Performance: On the Celeb-DF dataset, the RFM-enhanced Xception model achieved an AUC of 99.94% and a True Detect Rate (TDR) of 87.08% at a very strict 0.01% False Detect Rate.
- Robustness: In the "Less-Forgery" experiments, the RFM-trained model maintained high accuracy, while the baseline model's performance plummeted. This proves the RFM model learned a much broader set of features.
- Manipulation Fingerprints: The FAM visualizations revealed that different deepfake methods leave traces in different places. Two-stage methods leave artifacts primarily on facial boundaries, while one-stage GANs leave traces distributed across the entire face. RFM allows the model to capture both styles effectively.
- Architecture Agnostic: RFM is a training strategy that can be wrapped around any vanilla CNN (like Xception or EfficientNet) to achieve state-of-the-art results without modifying the network structure.
Conclusion and Future Work
RFM demonstrates that data augmentation shouldn't just be random; by making augmentation adversarial to the model's current attention, you force the model to become more sophisticated. While the method increases training time due to the extra forward and backward passes required for FAM, the gains in generalization are significant. Future work may extend RFM to video sequences to mine for temporal "flicker" artifacts that occur between frames.
Summary updated