Texture and artifact decomposition for improving generalization in deep-learning-based deepfake detection

Jie Gao, Marco Micheletto, Giulia Orrù, Sara Concas, Xiaoyi Feng, Gian Luca Marcialis, Fabio RoliEngineering Applications of AI 2024 · DOI

TL;DR

This paper introduces a novel deepfake detection framework called TAD that decomposes deepfake images into their constituent "texture" and "artifact" components. By analyzing these components separately and then fusing their insights, TAD significantly improves the model's ability to generalize to unseen deepfake manipulation techniques, a critical challenge in current detection methods.

What Problem Does This Paper Address?

The rapid advancement of deepfake technology poses a significant threat, making it increasingly difficult for automated detection systems to keep pace. While many deepfake detection methods focus on achieving high accuracy on known manipulation types, they often struggle to generalize to new or unseen techniques. This lack of generalization is a major hurdle, as deepfake creators constantly evolve their methods. The paper argues that existing methods often overlook the interplay between texture inconsistencies (natural patterns disrupted by manipulation) and artifacts (unintended distortions introduced during synthesis), leading to compromised performance.

Methods

The core of the proposed method, Texture and Artifact Detector (TAD), lies in its ability to decompose deepfake images into two distinct components:

  • Texture Model: This component focuses on identifying inconsistencies in the natural texture of faces. It employs a self-supervised learning strategy with deformable convolution stacks to create "soft masks." These masks help separate the "target" (manipulated) facial region from the "source" (background) context, allowing the model to learn texture differences. A texture decoder is used to reconstruct the image as a constraint, enforcing the extraction of meaningful texture features.
  • Artifact Model: This component utilizes a shallow network to specifically capture and identify artifacts introduced during the deepfake generation process. These artifacts can include pixel misalignments, compression artifacts, or other visual distortions.
  • Fusion Strategy: To leverage the complementary information from both texture and artifact analysis, TAD employs a fusion strategy. The paper explores two main architectures:
  • Single-model training strategy (S-TAD): This involves training separate texture and artifact models and then fusing their outputs.
  • Ensemble training strategy (E-TAD): This more advanced strategy trains specialized texture and artifact models for each known manipulation type and then fuses their outputs. This ensemble approach aims to further boost generalization capabilities.
  • Fusion Rules: Various fusion rules (average, maximum, minimum, MLP, and accuracy-based weighted average) are investigated to combine the scores from individual models.

Datasets

The primary dataset used for training and evaluating the models is FaceForensics++ (FF++). This dataset is a standard benchmark for deepfake detection and includes authentic videos and forged videos produced using five manipulation techniques: DeepFakes (DF), Face2Face (F2F), FaceSwap (FS), FaceShifter (FSh), and NeuralTextures (NT). The FF++ dataset is also available in different compression levels (c23 and c40), which are used to test the model's robustness to compression artifacts.

Additional datasets used for testing and cross-dataset evaluation include:

  • WildDF: A challenging dataset of real-world deepfake videos with varied backgrounds and manipulations.
  • DFDC (DeepFake Detection Challenge): A large-scale dataset from Facebook.
  • CelebDF (Version 2): A dataset featuring celebrities from public YouTube content.

Key Insights and Ideas

  • Decomposition is Key: The paper strongly advocates for decomposing deepfake detection into analyzing texture inconsistencies and artifacts separately. This separation allows for a more focused and effective analysis of each component.
  • Texture and Artifacts are Complementary: The synergy between texture and artifact information is crucial. While artifacts can sometimes obscure texture information, their combined analysis provides a more robust detection signal.
  • Self-Supervised Learning for Masks: The use of self-supervised learning to generate soft masks for separating foreground and background is a significant contribution, reducing reliance on pre-annotated masks which are often unavailable in real-world datasets.
  • Ensemble Learning for Generalization: The ensemble training strategy, where specialized models are trained for different manipulation types and then fused, proves to be highly effective in improving generalization to unseen deepfakes.

Key Results

  • Improved Generalization: The proposed TAD framework, particularly with the ensemble training strategy (E-TAD), demonstrates significantly improved generalization capabilities compared to several state-of-the-art (SOTA) methods, especially in cross-manipulation and cross-dataset scenarios.
  • Strong Intra-Dataset Performance: In intra-dataset evaluations on FF++, E-TAD achieves high accuracies, often surpassing SOTA methods, with accuracies reaching up to 97.11% for DF.
  • Robustness to Compression: The E-TAD model shows good performance even on heavily compressed videos (c40), indicating resilience to common real-world degradation.
  • Fusion Benefits: Ablation studies confirm that fusing texture and artifact information consistently outperforms individual texture-only or artifact-only models. The MLP fusion rule generally performs best.

Limitations and Future Work

  • Low-Light and Heavily Compressed Videos: The authors acknowledge a noticeable drop in accuracy for E-TAD in low-light conditions and with heavily compressed videos.
  • Cross-Format Generalization: A limitation identified is the difficulty in generalizing across different video formats, which leads to a significant drop in accuracy between intra-dataset and cross-dataset results.
  • Future Work: The authors propose future work to address these limitations by:
  • Integrating data augmentation techniques that simulate a wider range of lighting and compression effects.
  • Exploring adaptive neural network architectures that dynamically adjust to varying image qualities.
  • Investigating methods to improve generalization across heterogeneous video formats.

Summary updated

BibTeX
@article{gao2024texture,
  author = {Jie Gao and Marco Micheletto and Giulia Orrù and Sara Concas and Xiaoyi Feng and Gian Luca Marcialis and Fabio Roli},
  title = {{Texture and artifact decomposition for improving generalization in deep-learning-based deepfake detection}},
  journal = {Engineering Applications of AI},
  year = {2024},
  doi = {10.1016/j.engappai.2024.108450},
}
Figure