Exposing Deepfake Face Forgeries With Guided Residuals

Zhiqing Guo, Gaobo Yang, Jiyou Chen, Xingming Sun

TL;DR

This paper introduces GRnet, a novel deepfake detection framework that effectively fuses spatial and residual image features. By employing a "guided residuals" approach to extract manipulation traces and an attention mechanism to adaptively weigh features, GRnet significantly improves detection accuracy, especially on challenging low-quality images.

What Problem Does This Paper Address?

The proliferation of deepfake technology poses a significant threat to the credibility of visual content, with applications ranging from misinformation to identity theft. While existing deepfake detection methods often rely on either spatial features (like textures and shapes) or residual features (differences between an image and its predicted content), these approaches have limitations. Spatial features can be too general, and residual features, often derived from prediction models, can be prone to bias and degraded by common image processing operations like compression. This paper addresses the challenge of robustly detecting deepfakes, particularly in real-world scenarios where images may have low quality due to compression or other artifacts, by proposing a method that leverages the complementary strengths of both spatial and residual information.

Methods

The core of the proposed GRnet is a dual-stream architecture that processes both the original RGB image (spatial stream) and a "guided residual" image (residual stream).

  • Manipulation Trace Extractor (MTE): Instead of relying on prediction-based methods to obtain residuals, the authors introduce a novel MTE. This module uses a guided filter to preserve image content while effectively suppressing it in flat regions, thereby highlighting subtle manipulation traces. The guided residuals are then obtained by subtracting the filtered content from the original image. This approach aims to avoid the potential bias introduced by prediction models.
  • Attention Fusion Mechanism (AFM): To effectively combine the features learned from the spatial and residual streams, an AFM is employed. This mechanism adaptively allocates weights to each stream based on their respective cross-entropy losses. It also incorporates channel attention to capture dependencies between feature channels, allowing the network to prioritize more discriminative features for different image qualities.
  • Dual-Stream Architecture: The RGB images and guided residual images are fed into separate backbone networks (pre-trained ResNet-18) for feature extraction. The outputs from these streams are then fused by the AFM before being passed to a final detector.

Datasets

The paper evaluates GRnet on four publicly available datasets:

  • Hybrid Fake Face (HFF): Contains a large number of real and fake face images generated by various forgery techniques.
  • FaceForensics++ (FF): A dataset of video sequences manipulated with four common facial forgery techniques, provided in both high-quality (HQ) and low-quality (LQ) versions.
  • DeepFake Detection Challenge (DFDC): A dataset with real and fake video sequences, known for its challenging nature.
  • Celeb-DF: A recent, challenging dataset of hyper-realistic deepfake videos used for evaluating generalization ability.

Key Insights and Ideas

  • Guided Residuals for Unbiased Feature Extraction: The core idea of using a guided filter to extract residuals is a significant departure from prediction-based methods. This approach directly targets manipulation traces while minimizing content bias, leading to more reliable features.
  • Complementarity of Spatial and Residual Features: The paper strongly emphasizes that spatial and residual features are not mutually exclusive but rather complementary. Spatial features are more robust to low-quality images, while residual features excel at capturing subtle artifacts in high-quality images.
  • Adaptive Fusion is Crucial: The success of GRnet hinges on its ability to adaptively fuse features. The AFM's dynamic weighting of spatial and residual streams based on image quality is key to achieving robust performance across diverse scenarios.
  • Low-Quality Images Remain a Major Challenge: The paper highlights that even with advanced techniques, low-quality images (due to compression) significantly degrade manipulation traces, making detection harder. The proposed fusion mechanism aims to mitigate this.

Key Results

  • State-of-the-Art Performance: GRnet consistently outperforms existing state-of-the-art methods across multiple datasets and scenarios.
  • High Accuracy on HFF: On the HFF dataset, GRnet achieves an average accuracy of 98.53% for multiple classification and 97.72% for binary classification, significantly outperforming other methods.
  • Robustness to Compression: GRnet demonstrates strong robustness to image compression. For instance, on the HFF-JP60 dataset, it achieves an accuracy of 96.48%, a substantial improvement over previous methods.
  • Effective Feature Fusion: The AFM is shown to be superior to other common fusion methods (max, concatenation, sum, min), achieving the best accuracy on the compressed HFF dataset.
  • Generalization Ability: GRnet shows promising generalization capabilities, achieving the highest AUC score on the challenging Celeb-DF dataset when trained on the FF dataset.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, deep learning models with dual streams and attention mechanisms can be computationally intensive.
  • Reliance on Guided Filter: The effectiveness of the MTE is tied to the guided filter's ability to preserve content. If the content itself is highly manipulated or unusual, the residual extraction might be affected.
  • Future Work: The paper suggests that the proposed GRnet can be further improved by exploring more advanced attention mechanisms and backbone networks. The authors also imply that further research into handling even more challenging low-quality scenarios and diverse forgery types would be beneficial.

Summary updated

Abstract

For Deepfake detection, residual-based features can preserve tampering traces and suppress irrelevant image content. However, inappropriate residual prediction brings side effects on detection accuracy. Meanwhile, residual-domain features are easily affected by some image operations such as lossy compression. Most existing works exploit either spatial-domain or residual-domain features, which are fed into the backbone network for feature learning. Actually, both types of features are mutually correlated. In this work, we propose an adaptive fusion based guided residuals network (AdapGRnet), which fuses spatial-domain and residual-domain features in a mutually reinforcing way, for Deepfake detection. Specifically, we present a fine-grained manipulation trace extractor (MTE), which is a key module of AdapGRnet. Compared with the prediction-based residuals, MTE can avoid the potential bias caused by inappropriate prediction. Moreover, an attention fusion mechanism (AFM) is designed to selectively emphasize feature channel maps and adaptively allocate the weights for two streams. Experimental results show that AdapGRnet achieves better detection accuracies than the state-of-the-art works on four public fake face datasets including HFF, FaceForensics++, DFDC and CelebDF. Especially, AdapGRnet achieves an accuracy up to 96.52% on the HFF-JP60 dataset, which improves about 5.50%. That is, AdapGRnet achieves better robustness than the existing works.
BibTeX
@article{guo2023exposing,
  author = {Zhiqing Guo and Gaobo Yang and Jiyou Chen and Xingming Sun},
  title = {{Exposing Deepfake Face Forgeries With Guided Residuals}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=jAz4p8bxXk},
}
Figure