Fake Face Detection via Adaptive Residuals Extraction Network

Zhiqing Guo, Gaobo Yang, Jiyou Chen, Xingming SunarXiv 2020 · OpenReview

TL;DR

This paper introduces AMTENnet, a novel deep learning architecture for detecting manipulated face images. AMTENnet employs an "Adaptive Manipulation Traces Extraction Network" (AMTEN) to specifically highlight subtle artifacts left by face manipulation techniques, outperforming existing methods by achieving up to 98.52% accuracy on diverse datasets and demonstrating robustness against various post-processing operations.

What Problem Does This Paper Address?

The internet is increasingly flooded with sophisticated fake face images generated by techniques like Face2Face and Deepfake. While existing methods can detect specific manipulations, they struggle with complex scenarios involving compression, blurring, or scaling. Convolutional Neural Networks (CNNs), while powerful, tend to focus on image content rather than the subtle traces left by manipulation. This paper addresses the critical need for a robust, general-purpose fake face detector that can effectively identify forgeries even under challenging real-world conditions and unknown post-processing steps.

Methods

The core innovation is the Adaptive Manipulation Traces Extraction Network (AMTEN). Unlike traditional CNNs that learn content, AMTEN is designed as a pre-processing module to suppress image content and amplify manipulation artifacts. It uses an adaptive convolution layer that learns to predict manipulation traces during back-propagation. These traces are then "reused" in subsequent layers to maximize their impact.

The overall architecture, AMTENnet, integrates AMTEN with a standard CNN. The process involves:

  1. AMTEN Module: A first convolutional layer (Conv 1) predicts pixel values. The manipulation traces (Fmt) are then extracted by subtracting the original image from this predicted feature map. This adaptive extraction is key, as it learns traces specific to the input.
  2. Hierarchical Feature Extraction (HFE): The extracted traces (Fmt) are passed through further convolutional layers, along with the original traces, to create "stable manipulation traces" (Freu). This hierarchical approach allows for the extraction of increasingly complex and robust forensic features.
  3. Classification: The learned features are fed into fully connected layers and a softmax classifier to determine if the image is real or fake.

The paper also highlights the importance of feature reuse within AMTEN to prevent information loss and improve the learning of manipulation traces.

Datasets

The authors introduce a Hybrid Fake Face (HFF) dataset for their experiments. This dataset comprises:

  • Real Face Images: Sourced from CelebA (low-resolution), CelebA-HQ (high-resolution), and YouTube-Frame (face video frames) to simulate real-world internet conditions.
  • Fake Face Images: Generated using representative Face Image Manipulation (FIM) techniques, including:
  • Identity Manipulation: PGGAN, StyleGAN
    • Expression Manipulation: Face2Face, Glow
    • Attribute Transfer: StarGAN (with five attribute variations)

Additionally, the FaceForensics++ (FF++) dataset is used for experiments, which includes videos manipulated by Deepfakes, Face2Face, FaceSwap, and NeuralTextures, further compressed with high (HQ) and low (LQ) quality levels.

Key Insights and Ideas

  • Focusing on Traces, Not Content: The fundamental idea is that FIM leaves subtle, often imperceptible, "traces" that are distinct from natural image content. AMTEN's design prioritizes learning these traces.
  • Adaptive Learning of Traces: Instead of using fixed filters or hand-crafted features, AMTEN adaptively learns manipulation traces during training, making it more flexible and effective against a wider range of manipulations.
  • Feature Reuse for Robustness: Reusing the extracted manipulation traces in later layers helps to preserve and amplify these subtle artifacts, leading to more discriminative features.
  • Simulating Real-World Complexity: The authors go beyond simple fake/real classification by incorporating post-processing operations (compression, blurring, scaling) to create more realistic testing scenarios, demonstrating the model's practical applicability.
  • The Power of Mixed Data Training: Training the detector on datasets that have undergone various mixed post-processing operations significantly improves its generalization capability to unseen manipulations.

Key Results

  • High Accuracy: AMTENnet achieves an average accuracy of 98.52% when detecting fake face images generated by various FIM techniques.
  • Robustness to Post-Processing: The detector achieves an average accuracy of 95.17% when detecting face images with unknown post-processing operations.
  • Outperforms Baselines: AMTENnet significantly outperforms existing state-of-the-art methods, including Meso-4, MesoInception-4, Hand-Crafted-Res, and MISLnet, by a notable margin. For instance, it improves average detection accuracy by about 7.61% compared to MISLnet.
  • Effective AMTEN Module: Experiments show that removing AMTEN from AMTENnet reduces accuracy by 2.36%, highlighting AMTEN's crucial role in extracting effective manipulation traces.
  • Generalization: Training with mixed post-processing data (e.g., JP-mix, ME-mix) leads to improved generalization, with an average accuracy of 95.17% on unseen operations.

Limitations and Future Work

  • Simulated Scenarios: While the authors simulate complex scenarios using post-processing, they acknowledge that practical forensics cases might involve more intricate manipulations.
  • Future Work: The authors suggest further improving robustness by gathering real-world samples from social media platforms. They also propose AMTEN as a potential general-purpose residual predictor for other face forensic tasks.

Summary updated

Abstract

With the proliferation of face image manipulation (FIM) techniques such as Face2Face and Deepfake, more fake face images are spreading over the internet, which brings serious challenges to public confidence. Face image forgery detection has made considerable progresses in exposing specific FIM, but it is still in scarcity of a robust fake face detector to expose face image forgeries under complex scenarios such as with further compression, blurring, scaling, etc. Due to the relatively fixed structure, convolutional neural network (CNN) tends to learn image content representations. However, CNN should learn subtle manipulation traces for image forensics tasks. Thus, we propose an adaptive manipulation traces extraction network (AMTEN), which serves as pre-processing to suppress image content and highlight manipulation traces. AMTEN exploits an adaptive convolution layer to predict manipulation traces in the image, which are reused in subsequent layers to maximize manipulation artifacts by updating weights during the back-propagation pass. A fake face detector, namely AMTENnet, is constructed by integrating AMTEN with CNN. Experimental results prove that the proposed AMTEN achieves desirable pre-processing. When detecting fake face images generated by various FIM techniques, AMTENnet achieves an average accuracy up to 98.52%, which outperforms the state-of-the-art works. When detecting face images with unknown post-processing operations, the detector also achieves an average accuracy of 95.17%.
BibTeX
@misc{guo2020fake,
  author = {Zhiqing Guo and Gaobo Yang and Jiyou Chen and Xingming Sun},
  title = {{Fake Face Detection via Adaptive Residuals Extraction Network}},
  year = {2020},
  eprint = {2005.04945},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.04945},
}
Figure