Domain General Face Forgery Detection by Learning to Weight

Ke Sun, Hong Liu, Qixiang Ye, Yue Gao, Jianzhuang Liu, Ling Shao, Rongrong Ji

Introduction

Deepfake detection has become a critical area of research as face manipulation techniques grow increasingly sophisticated. While existing detectors perform well on datasets they were trained on, they often fail to generalize to "unseen" domains—new manipulation methods or datasets with different characteristics. This lack of robustness stems from models overfitting to domain-specific artifacts rather than learning universal features of forgery.

To address this, we present Learning-to-Weight (LTW), a meta-learning framework designed to enhance domain generalization. By automatically assigning importance to training samples based on their contribution to generalization, LTW builds a more robust detector capable of identifying novel forgeries.

The Learning-to-Weight (LTW) Framework

The core challenge in face forgery detection is the "Deepfake arms race," where attackers constantly invent new methods like identity swapping, expression transfer, or neural textures. Standard models often capture the specific "fingerprints" of these generators, leading to poor performance on unseen data.

The LTW framework addresses this by treating training as a meta-learning problem. It consists of a Detector Branch (a binary classifier, typically using EfficientNet-b0) and a Weight-Aware Branch. The latter is a lightweight network that predicts an importance score (weight) for each training sample based on its latent features.

Overview of the LTW framework, illustrating the Meta-Split strategy and the transition from training to testing on unseen datasets.

Meta-Split Strategy

To simulate the encounter with unseen domains during training, LTW employs a Meta-Split Strategy. Multi-source training data is randomly partitioned into a "training set" and a "meta-test set." This allows the model to "practice" generalization by evaluating how updates performed on the training set affect performance on the meta-test set.

Meta-Optimization and Loss Functions

The training process involves a sophisticated meta-optimization loop. First, the model performs a "virtual" update on the training set. It then evaluates the loss on the meta-test set. If a particular training sample helps the model perform better on the "unseen" meta-test set, the weight-aware branch learns to assign higher weights to similar samples in future iterations. This effectively teaches the model to ignore domain-specific noise and focus on domain-invariant features.

Detailed view of the Meta-Training step, showing the interaction between the training set, meta-test set, and the weight-aware branch.

Beyond standard cross-entropy (CE) loss, LTW introduces the Intra-Class Compact (ICC) Loss. This loss function is based on the observation that "real faces" form a stable, consistent category, whereas "fake faces" are infinitely diverse. ICC loss pulls all real samples toward a single center in the feature space while pushing fake samples away, creating a tight, reliable boundary for authenticity.

Experimental Evaluation

The LTW method was evaluated against prominent benchmarks, including FaceForensics++ (FF++), Celeb-DF, and the Deepfake Detection Challenge (DFDC). Two primary setups were tested:

  1. General Intra-Datasets (GID): Training on three manipulation types within FF++ and testing on the fourth.
  2. General Cross-Datasets (GCD): Training on FF++ and testing on entirely different datasets like Celeb-DF and DFDC.

Key Results

LTW consistently outperformed state-of-the-art baselines such as MLDG and ForensicTransfer. Key findings include:

  • Superior Generalization: LTW showed an average improvement of approximately 2% across all benchmarks.
  • Robustness to Compression: On low-quality, compressed video (GID-F2F40), LTW achieved an AUC of 0.724, significantly higher than the baseline's 0.537.
  • Cross-Dataset Success: When tested on the challenging Celeb-DF dataset after training on FF++, LTW reached an accuracy of 63.4%, demonstrating its effectiveness in real-world, unseen scenarios.

Conclusion and Future Work

By learning what to ignore, the LTW framework shifts the focus from learning specific forgery artifacts to identifying universal characteristics of real and fake faces. While meta-learning introduces higher computational costs during training due to second-order derivatives, the resulting models are significantly more robust. Future research may explore the resilience of this weighting strategy against adversarial attacks specifically designed to bypass deepfake detectors.

Summary updated

Abstract

In this paper, we propose a domain-general model, termed learning-to-weight (LTW), that guarantees face detection performance across multiple domains, particularly the target domains that are never seen before. However, various face forgery methods cause complex and biased data distributions, making it challenging to detect fake faces in unseen domains. We argue that different faces contribute differently to a detection model trained on multiple domains, making the model likely to fit domain-specific biases. As such, we propose the LTW approach based on the meta-weight learning algorithm, which configures different weights for face images from different domains. The LTW network can balance the model's generalizability across multiple domains. Then, the meta-optimization calibrates the source domain's gradient enabling more discriminative features to be learned. The detection ability of the network is further improved by introducing an intra-class compact loss. Extensive experiments on several commonly used deepfake datasets to demonstrate the effectiveness of our method in detecting synthetic faces. Code and supplemental material are available at https://github.com/skJack/LTW.
BibTeX
@inproceedings{sun2021domain,
  author = {Ke Sun and Hong Liu and Qixiang Ye and Yue Gao and Jianzhuang Liu and Ling Shao and Rongrong Ji},
  title = {{Domain General Face Forgery Detection by Learning to Weight}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i3.16367},
}
Figure