DeepFeatureX Net: Deep Features eXtractors based Network for discriminating synthetic from real images

Orazio Pontorno, Luca Guarnera, Sebastiano BattiatoarXiv 2024 · arXiv

TL;DR

DeepFeatureX Net is a novel deepfake detection framework that uses specialized "Base Models" trained on highly unbalanced datasets to extract unique architectural fingerprints from GANs, Diffusion Models, and real images. By focusing on these specific class-based features, the system achieves superior generalization to unseen AI models and maintains high accuracy even under heavy JPEG compression.

What Problem Does This Paper Address?

As generative AI evolves, the "cat-and-mouse" game of deepfake detection becomes increasingly difficult. Current state-of-the-art (SOTA) detectors often suffer from a generalization gap: they perform exceptionally well on the specific AI architectures they were trained on (e.g., ProGAN) but fail miserably when encountering images from a new or unseen model (e.g., Stable Diffusion).

Furthermore, most detectors rely on subtle high-frequency artifacts that are easily erased by JPEG compression, the standard processing step for almost every social media platform. The authors aim to build a detector that is both robust to compression and capable of identifying "fakes" from architectures it has never seen before.

Methods

The core innovation of DeepFeatureX Net is its multi-backbone feature extraction strategy. Instead of training one model to do everything, the authors split the task into three specialized blocks:

  1. Base Model Training: The researchers use three standard CNN backbones (like DenseNet or ResNet). Each is trained on a deliberately unbalanced dataset (a 90:10 ratio). For example, the "GAN Base Model" is trained on 90% GAN images and 10% "others" (Real/DM). This forces the model to maximize recall for its specific class, effectively learning the unique "fingerprint" of that generation family.
  2. Feature Extraction & Fusion: Once trained, the Base Models are frozen. The final classification layer is removed, and the raw feature maps from all three models are concatenated into a single tensor.
  3. Classification: This combined feature set is processed by a custom 1D Convolutional Neural Network followed by Global Average Pooling (GAP) and a Fully Connected (FC) layer to produce the final classification: Real, GAN, or Diffusion Model.
The entire pipeline of the proposed method, showing the unbalanced training phase (a) and the final architecture (b).

Datasets

The study utilized a massive dataset of 72,334 images to ensure variety in semantics and resolution:

  • Real Images: Sourced from CelebA, FFHQ, AFHQ, ImageNet, and COCO.
  • GAN-Generated: Images from 11 different architectures, including GauGAN, BigGAN, StyleGAN (1, 2, and 3), and CycleGAN.
  • Diffusion Model (DM)-Generated: Images from DALL-E MINI, DALL-E 2, Latent Diffusion, Stable Diffusion, and Glide.

For evaluation, they created specific "Generalization Sets" categorized as $T^i$ (seen models), $T^o$ (unseen models), and $T^{i/o}$ (mixed).

Key Insights and Ideas

  • Imbalance as a Feature, Not a Bug: Usually, researchers try to balance datasets. Here, the authors use extreme imbalance to "force" the backbones to become specialists. This specialization allows the models to capture intrinsic architectural traces rather than just general image statistics.
  • High Recall is the Goal: By prioritizing recall in the Base Models (often reaching >99%), the system ensures that if a specific type of artifact exists, at least one of the specialized backbones will catch it.
  • DenseNet Dominance: The authors found that models from the DenseNet family consistently provided the best boost to overall performance compared to ResNet or ViT (Vision Transformer) backbones.

Key Results

  • High Accuracy: In a three-class classification task (Real vs. GAN vs. DM), the model achieved over 93% accuracy using a DenseNet-161 backbone.
  • JPEG Robustness: While performance naturally drops as compression increases, the model remained remarkably stable. Even at a Quality Factor (QF) of 50—where JPEG blocks become clearly visible—the binary detection accuracy (Deepfake vs. Real) remained high.
  • Superior Generalization: In tests against unseen models, DeepFeatureX Net outperformed SOTA methods by over 10% in many contexts. While other detectors' performance "dropped drastically" on new architectures, this approach maintained its discriminative power.
Visual representation of JPEG compression levels (Raw, QF80, QF50) used to test model robustness.

Limitations and Future Work

The authors acknowledge that while their three-block approach (GAN, DM, Real) is effective for current technologies, the landscape of generative AI is expanding. Future work may need to incorporate more specialized blocks for other types of generation, such as Variational Autoencoders (VAEs).

Additionally, the researchers plan to explore spatial analysis of the feature extractor outputs and develop "structure-independent" architectures to further improve the system's ability to handle the rapid evolution of deepfake generation techniques.

Summary updated

Abstract

Deepfakes, synthetic images generated by deep learning algorithms, represent one of the biggest challenges in the field of Digital Forensics. The scientific community is working to develop approaches that can discriminate the origin of digital images (real or AI-generated). However, these methodologies face the challenge of generalization, that is, the ability to discern the nature of an image even if it is generated by an architecture not seen during training. This usually leads to a drop in performance. In this context, we propose a novel approach based on three blocks called Base Models, each of which is responsible for extracting the discriminative features of a specific image class (Diffusion Model-generated, GAN-generated, or real) as it is trained by exploiting deliberately unbalanced datasets. The features extracted from each block are then concatenated and processed to discriminate the origin of the input image. Experimental results showed that this approach not only demonstrates good robust capabilities to JPEG compression but also outperforms state-of-the-art methods in several generalization tests. Code, models and dataset are available at https://github.com/opontorno/block-based_deepfake-detection.
BibTeX
@misc{pontorno2024deepfeaturex,
  author = {Orazio Pontorno and Luca Guarnera and Sebastiano Battiato},
  title = {{DeepFeatureX Net: Deep Features eXtractors based Network for discriminating synthetic from real images}},
  year = {2024},
  eprint = {2404.15697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.15697},
}
Figure