Rethinking Gradient Operator for Exposing AI-enabled Face Forgeries

Zhiqing Guo, Gaobo Yang, Dengyong Zhang, Ming XiaExpert Systems with Applications 2022 · OpenReview

TL;DR

This paper introduces GocNet, a novel deep learning framework for detecting AI-generated (deepfake) faces. It leverages gradient operators within two new plug-and-play modules, TP and MTA, to effectively highlight subtle manipulation traces that standard CNNs often miss, leading to significantly improved detection accuracy.

What Problem Does This Paper Address?

The proliferation of AI-generated faces, or "deepfakes," poses a serious threat to trust and authenticity in digital media. While Convolutional Neural Networks (CNNs) are powerful tools for image analysis, they tend to focus on the semantic content of faces rather than the subtle artifacts left by forgery techniques. Existing methods often rely on subtracting predicted pixel values from original ones to emphasize these traces, but this can lead to performance degradation due to complex learning mechanisms. The paper aims to bridge this gap by developing a more effective and interpretable approach to highlight manipulation traces for robust face forgery detection.

Methods

The core innovation lies in two novel modules:

  • Tensor Pre-processing (TP) Module: This module embeds classical gradient operators (like Sobel, Prewitt, Laplacian) directly into the CNN as fixed convolution layers. Instead of learning complex transformations, it uses these operators to refine feature tensors, accentuating the subtle changes in image gradients that signal manipulation, while downplaying irrelevant content. This is an "end-to-end" approach, avoiding the time-consuming offline pre-processing of previous methods.
  • Manipulation Trace Attention (MTA) Module: This module further refines features by focusing on manipulation traces. It operates in two dimensions:
  • Double Pooling (DP) Gate: Learns channel-wise attention by pooling spatial information to capture manipulation trace distributions.
  • Manipulation Trace (MT) Gate: Utilizes a gradient operator (similar to the TP module) to refine feature maps, emphasizing manipulation traces and filtering out irrelevant content.

These modules are integrated into a dual-stream architecture called GocNet. One stream uses the TP module for initial pre-processing, while the other embeds the MTA module within the backbone network (ResNet-18) to continuously focus on manipulation traces. The features from both streams are fused for final classification.

Datasets

The proposed GocNet was evaluated on five benchmark datasets:

  • FaceForensics++ (FF++): A large-scale dataset with real and fake video sequences manipulated by four methods, offering different quality levels (HQ and LQ).
  • Deep Fake Detection (DFD): A dataset from Google & JigSaw containing deepfake video sequences.
  • DeepFake Detection Challenge (DFDC): Another large-scale video dataset with real and fake sequences.
  • Hybrid Fake Face (HFF): A large-scale image dataset featuring real images from CelebA and fake images generated by various GANs (Glow, Face2Face, StarGAN, PGGAN, StyleGAN).
  • Celeb-DF (CDF): A recent, challenging dataset of high-quality deepfake video sequences designed to reduce the visual gap with real videos.

Key Insights and Ideas

  • Gradient Operators are Powerful for Forgery Detection: The paper strongly advocates for the under-appreciated power of classical gradient operators in revealing subtle manipulation traces that are invisible to the naked eye and often missed by standard CNNs.
  • Fixed Kernels vs. Learned Kernels: By using fixed gradient kernels in the TP module, the authors avoid the potential for learned kernels to over-complicate the process or inadvertently destroy manipulation traces, leading to a simpler and more effective pre-processing step.
  • Dual-Stream Architecture for Complementary Focus: The dual-stream approach effectively separates the initial trace highlighting (TP) from the continuous attention to traces within the network (MTA), preventing potential interference between the two processes and leading to superior performance.
  • Attention Beyond Spatial/Channel: The MTA module's innovation lies in explicitly designing an attention mechanism that focuses on manipulation traces themselves, rather than just general spatial or channel interdependencies.

Key Results

GocNet consistently outperformed prior state-of-the-art methods across all five tested datasets.

  • The TP module alone, when used with the Prewitt operator in the diagonal direction, improved accuracy by at least 4.60% compared to existing pre-processing modules.
  • The full GocNet (dual-stream with TP and MTA) achieved the best results, for example, on the FF++ dataset with C40 compression, it reached an accuracy of 94.21% and an EER of 0.1379.
  • The paper demonstrates that the proposed modules are more effective than other attention mechanisms like CBAM and DA, with BaseNet-MTA achieving an accuracy of 86.61% and AUC of 91.47%.

Limitations and Future Work

  • Computational Cost: While the TP module is efficient, the dual-stream architecture and MTA module likely increase computational demands compared to single-stream, non-attention-based models. The paper mentions that complex CNN structures like XceptionNet achieve competitive results but require significant training time, implying a trade-off.
  • Generalization to New Forgery Techniques: While tested on diverse datasets, the generalization to entirely novel or unseen manipulation techniques remains an ongoing challenge in the field.
  • Further Exploration of Gradient Operators: The paper explored nine gradient operators for the TP module, with Prewitt in the diagonal direction showing the best results. Future work could explore more advanced or custom gradient-based operators.
  • Adaptive Coefficient in MTA: The MTA module uses an adaptive coefficient α for combining attention maps. Further research could investigate more sophisticated methods for determining or learning this coefficient.

Summary updated

Abstract

For image forensics, convolutional neural networks (CNNs) tend to learn content features rather than subtle manipulation traces, which limits forensic performance. Existing methods predominantly solve the above challenges by following a general pipeline, that is, subtracting the original pixel value from the predicted pixel value to make CNNs pay attention to the manipulation traces. However, due to the complicated learning mechanism, these methods may bring some unnecessary performance losses. In this work, we rethink the advantages of gradient operator in exposing face forgery, and design two plug-and-play modules by combining gradient operator with CNNs, namely tensor pre-processing (TP) and manipulation trace attention (MTA) module. Specifically, TP module refines the feature tensor of each channel in the network by gradient operator to highlight the manipulation traces and improve the feature representation. Moreover, MTA module considers two dimensions, namely channel and manipulation traces, to force the network to learn the distribution of manipulation traces. These two modules can be seamlessly integrated into CNNs for end-to-end training. Experiments show that the proposed network achieves better results than prior works on five public datasets. Especially, TP module greatly improves the accuracy by at least 4.60% compared with the existing pre-processing module only via simple tensor refinement. The code is available at: https://github.com/EricGzq/GocNet-pytorch.
BibTeX
@article{guo2022rethinking,
  author = {Zhiqing Guo and Gaobo Yang and Dengyong Zhang and Ming Xia},
  title = {{Rethinking Gradient Operator for Exposing AI-enabled Face Forgeries}},
  journal = {Expert Systems with Applications},
  year = {2022},
  url = {https://openreview.net/forum?id=WxMUIq9lqA},
}
Figure