Towards Intrinsic Common Discriminative Features Learning for Face Forgery Detection Using Adversarial Learning

Wanyi Zhuang, Qi Chu, Haojie Yuan, Changtao Miao, Bin Liu, Nenghai Yu

TL;DR

Researchers from USTC have developed a new face forgery detection method that uses adversarial learning to "strip away" irrelevant information like facial identity and specific manipulation techniques. By forcing the model to ignore these "distractions," it learns intrinsic features that allow it to detect deepfakes much more effectively across different datasets and low-quality videos.

What Problem Does This Paper Address?

Most deepfake detectors are trained as simple binary classifiers: "Is this image real or fake?" While this works well on the specific dataset used for training, these models often fail in the real world.

The authors observed a critical flaw: standard neural networks (like Xception) don't just learn what a "fake" looks like. Instead, they inadvertently learn to cluster images based on the forgery method (e.g., "this looks like a Deepfakes-style artifact") or the facial identity (e.g., "this is a feature of Person A"). When the model encounters a new person or a brand-new forgery technique, it gets confused because the features it relies on are no longer present. The goal of this paper is to force the model to learn "intrinsic common discriminative features"—the fundamental essence of a forgery that remains constant regardless of who is in the video or how it was made.

Methods

The authors propose a multi-task adversarial learning framework consisting of three main components:

  1. Feature Generator & Binary Classifier: The core "detector" that tries to distinguish real faces from fake ones.
  2. Forgery Method Discriminator: This component tries to guess which specific manipulation tool was used (e.g., FaceSwap vs. NeuralTextures).
  3. Identity Discriminator: This component tries to distinguish between different individuals.

The "secret sauce" is the Gradient Reverse Layer (GRL). During training, the Feature Generator tries to minimize the detection loss but maximize the loss of the two discriminators. Essentially, the generator is punished if the discriminators can tell which forgery method was used or who the person is. This "adversarial" pressure forces the generator to discard identity and method-specific textures, leaving only the intrinsic "fake-ness" behind.

Handling Missing Data: Since many deepfake datasets don't provide ground-truth identity labels for every face, the authors designed a clever Identity Similarity Supervision module. They use a pre-trained face recognition model (ArcFace) to calculate the similarity between faces and use those scores to guide the identity discriminator, ensuring the model ignores identity even without explicit labels.

Datasets

The researchers evaluated their method on several industry-standard benchmarks:

  • FaceForensics++ (FF++): A massive dataset containing four different forgery methods at various compression levels (Raw, High Quality, and Low Quality).
  • Celeb-DF (v2): A challenging dataset known for high-quality, realistic forgeries, used here primarily for cross-dataset generalization testing.
  • DeepFakeDetection (DFD): Used to validate the identity-adversarial components.

Key Insights and Ideas

  • The "Aha!" Moment (t-SNE Visualization): The paper provides striking visualizations showing that in vanilla models, "real" and "fake" features are often tangled up with identity clusters. After applying their adversarial method, the features for "real" and "fake" become much more distinct and compact, regardless of the person's face.
  • Identity is a Distraction: The model proves that facial identity is actually a "noisy" signal for forgery detection. By intentionally making the model "blind" to who the person is, it becomes a better judge of whether the pixels have been tampered with.
  • Robustness to Compression: One of the hardest parts of deepfake detection is "Low Quality" (LQ) video where compression wipes out fine details. This method shows significant gains in LQ scenarios, suggesting that intrinsic features are more robust than high-frequency textures.

Key Results

  • Cross-Dataset Generalization: When trained on FF++ and tested on the unseen Celeb-DF dataset, the proposed method achieved an AUC of 79.69%, significantly outperforming standard Xception (68.48%) and other state-of-the-art models.
  • Low-Quality Performance: On the FF++ Low Quality (LQ) set, the model achieved 94.74% AUC, outperforming several methods that specifically use extra frequency-domain information.
  • Unseen Forgery Detection: In "leave-one-out" tests (training on three methods and testing on a fourth), the model consistently beat the baseline, proving it can catch new types of forgeries it hasn't seen before.

Limitations and Future Work

  • Frame-Based Approach: The current model analyzes videos frame-by-frame. It does not yet incorporate temporal information (flickering or inconsistencies between frames), which is a common giveaway in deepfakes. The authors note that a video-based method (like Two-branch) still holds an edge in some specific cross-dataset tests.
  • Reliance on Pre-trained Models: The identity supervision relies on the quality of an off-the-shelf face recognition model (ArcFace). If the face recognition model fails (e.g., due to extreme angles or lighting), the identity discriminator's effectiveness might decrease.
  • Future Direction: Integrating this adversarial feature learning with temporal sequence modeling (like LSTMs or Transformers) could likely push the accuracy even higher.

Summary updated

Abstract

Existing face forgery detection methods usually treat face forgery detection as a binary classification problem and adopt deep convolution neural networks to learn discriminative features. The ideal discriminative features should be only related to the real/fake labels of facial images. However, we observe that the features learned by vanilla classification networks are correlated to unnecessary properties, such as forgery methods and facial identities. Such phenomenon would limit forgery detection performance especially for the generalization ability. Motivated by this, we propose a novel method which utilizes adversarial learning to eliminate the negative effect of different forgery methods and facial identities, which helps classification network to learn intrinsic common discriminative features for face forgery detection. To leverage data lacking ground truth label of facial identities, we design a special identity discriminator based on similarity information derived from off-the-shelf face recognition model. Extensive experiments demonstrate the effectiveness of the proposed method under both intra-dataset and cross-dataset evaluation settings.
BibTeX
@inproceedings{zhuang2022towards,
  author = {Wanyi Zhuang and Qi Chu and Haojie Yuan and Changtao Miao and Bin Liu and Nenghai Yu},
  title = {{Towards Intrinsic Common Discriminative Features Learning for Face Forgery Detection Using Adversarial Learning}},
  booktitle = {ICME},
  year = {2022},
  url = {https://openreview.net/forum?id=UbvVanIFCj},
}

Loading related content...

Figure