Open Set Classification of GAN-based Image Manipulations via a ViT-based Hybrid Architecture

Jun Wang, Omran Alamayreh, Benedetta Tondi, Mauro BarniWorkshops @ CVPR 2023 · DOI · arXiv

Introduction

In the rapidly evolving landscape of generative adversarial networks (GANs), deepfake detection has become a critical security challenge. Most existing detection tools operate in a closed-set environment, where the model is trained to recognize a fixed set of manipulation types (e.g., "smiling" or "aging"). However, when these models encounter a novel, unseen manipulation—such as a new hairstyle filter or a different generative architecture—they often incorrectly categorize it into one of the known classes with high confidence.

The Open Set Recognition (OSR) problem: identifying known manipulations while rejecting unknown ones.

To address this, researchers are shifting toward Open Set Recognition (OSR). A robust OSR system must accurately classify known manipulations while providing a "rejection option" for unknown inputs. This paper introduces a hybrid architecture that combines the strengths of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) to achieve state-of-the-art performance in both facial attribute editing classification and GAN attribution.

Proposed Hybrid Architecture

The core of the proposed method is a multi-branch architecture designed to capture both local forgery artifacts and global structural inconsistencies. Unlike standard pure-CNN approaches, this pipeline integrates a localization-guided branch to focus the model's attention on manipulated regions.

The proposed hybrid architecture featuring a ResNet50 backbone, a localization branch (FCN), and a classification branch (ViT).

1. Feature Extraction Backbone

The model uses a modified ResNet50 as its backbone. To preserve the fine-grained high-frequency details necessary for spotting subtle GAN artifacts, the authors removed the initial downsampling layers. This ensures that the resulting feature maps ($f_r$) retain sufficient spatial resolution for subsequent tasks.

2. The Localization Branch

Parallel to the classification task, a Fully Convolutional Network (FCN) head acts as a localization branch. During training, this branch is supervised to produce a binary mask of the manipulated area. While the mask itself is not required during inference, this auxiliary task forces the backbone to extract features specifically from the forged pixels, acting as a powerful attention mechanism that improves classification robustness.

3. The ViT Classification Module

Instead of processing raw image patches, the classification branch feeds the CNN-extracted feature maps into a Vision Transformer (ViT). The feature map is divided into patches ($f_p$), which are then converted into patch embeddings ($f_e$) and combined with position embeddings ($E_{pos}$). The ViT's self-attention mechanism allows the model to exploit long-range correlations across the image, which is particularly effective for identifying the global statistical signatures left by different GAN architectures.

Open Set Recognition and Rejection

To handle the "unknown" category, the authors explored several rejection strategies. The goal is to determine if the input belongs to a known class or should be rejected. The strategies tested include:

  • Maximum Softmax Probability (MSP): Using the highest probability from the final layer.
  • Maximum Logit Score (MLS): Using the raw output values before the Softmax layer.
  • OpenMax: A more complex statistical approach for OSR.

Interestingly, the Maximum Logit Score (MLS) consistently outperformed other methods. This suggests that the raw "energy" of the model's prediction is a more reliable indicator of whether a manipulation is "known" than the normalized probability distribution.

Experimental Results

The model was evaluated on two large-scale datasets: a GAN Editing Dataset (18 edit types via InterfaceGAN and StyleCLIP) and a GAN Attribution Dataset (50k images from five architectures including StyleGAN3 and Latent Diffusion).

Key findings include:

  • Superior Performance: The hybrid model achieved a closed-set accuracy of 94.34% and an open-set Area Under the Curve (AUC) of 91.98%, significantly outperforming existing OSR methods like ARPL and GCPL.
  • Ablation Insights: The inclusion of the ViT module and the localization branch provided a performance boost of approximately 10% in accuracy and 9% in AUC compared to a standard ResNet50 baseline.
  • Generalization: By forcing the model to "look" at manipulated pixels through localization guidance, it learned more discriminative features that generalized better to unseen generative models.

Conclusion and Future Work

This research demonstrates that combining CNN-based feature extraction with ViT-based global reasoning—guided by localization supervision—creates a highly effective system for open-set GAN detection. While the results are promising for facial imagery, future work will focus on improving robustness against post-processing attacks (like JPEG compression) and extending the methodology to diverse image domains beyond human faces.

Summary updated

BibTeX
@inproceedings{wang2023open,
  author = {Jun Wang and Omran Alamayreh and Benedetta Tondi and Mauro Barni},
  title = {{Open Set Classification of GAN-based Image Manipulations via a ViT-based Hybrid Architecture}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00102},
}
Figure