Deepfake Style Transfer Mixture: a First Forensic Ballistics Study on Synthetic Images

Luca Guarnera, Oliver Giudice, Sebastiano BattiatoarXiv 2022 · DOI · arXiv

TL;DR

This paper pioneers the field of "deepfake ballistics" by demonstrating that we can detect not just if an image is a deepfake, but how many times it has been processed by a style-transfer generative network. By analyzing frequency domain statistics and leveraging deep learning, the authors achieve up to 92.75% accuracy in distinguishing single-manipulated from double-manipulated synthetic faces, while also uncovering unique mathematical properties of the style-transfer operation itself.

What Problem Does This Paper Address?

Multimedia forensics has become highly proficient at distinguishing real images from deepfakes and identifying the specific Generative Adversarial Network (GAN) architecture used to create them. However, a major gap remains: reconstructing the history or lineage of a synthetic image—a field known as image ballistics.

If a malicious actor applies a style-transfer algorithm to a face once, and then applies another layer of manipulation later, can we detect this multi-stage processing? The authors address this challenge by investigating whether generative manipulations leave cumulative, detectable traces, treating the problem similarly to double-quantization detection in traditional JPEG forensics.

Fig. 1. Deepfake manipulation categories.

Methods

The authors propose two distinct approaches to classify whether an image has undergone a single style-transfer operation (termed Deepfake-2) or a double operation (Deepfake-3):

  1. Analytical Approach (Method 1): Following standard JPEG processing pipelines, images are split into $8 \times 8$ non-overlapping blocks, and the Discrete Cosine Transform (DCT) is applied. While the DC coefficient represents the average pixel value, the 63 AC coefficients are modeled using a zero-centered Laplacian distribution. The scale parameters ($\beta$) of these distributions form a 63-dimensional feature vector used to train traditional machine learning classifiers (e.g., Random Forest, SVM, k-NN).
  2. Deep Learning Approach (Method 2): A standard ResNet-18 architecture pre-trained on ImageNet is fine-tuned. The final fully-connected layer is modified to output a two-class distribution (single vs. double manipulation) followed by a SoftMax layer, optimized using Cross-Entropy loss.

Additionally, the paper formally tests the algebraic properties of the style-transfer operator ($\oplus$), evaluating Identity, Commutativity, and Associativity using Structural Similarity (SSIM) and RGB color histograms.

DCT block division and coefficient modeling

Datasets

The study utilizes images generated by StarGAN-V2, selected for its state-of-the-art ability to produce high-resolution, high-quality facial attribute transfers.

  • Deepfake-2 (Single Pass): 1,200 training and 200 testing images generated by transferring attributes between two distinct source datasets.
  • Deepfake-3 (Double Pass): 1,200 training and 200 testing images where the output of the first style-transfer pass is subjected to a second pass with a different target attribute.

In total, the benchmark dataset consists of 2,400 training images and 400 test images.

Key Insights and Ideas

  • Cumulative Fingerprints: Even when the exact same GAN architecture (StarGAN-V2) is used for both passes, the second operation does not erase the first trace. Instead, it creates a unique "combination" fingerprint that remains distinct from a single-pass image.
  • Asymmetry and Lack of Identity: Mathematically, style transfer behaves very differently from standard image operations. The authors prove that there is no true "neutral element" (Identity property fails)—even passing the exact same face as both source and target causes the GAN to reconstruct the image with subtle architectural artifacts.
  • Quasi-Associativity: While style transfer is strictly non-commutative ($A \oplus B \neq B \oplus A$), it exhibits quasi-associativity when looking at color distributions. The order of operations in a multi-stage pipeline, $((A \oplus B) \oplus C)$, yields an RGB histogram nearly identical to $(A \oplus (B \oplus C))$, providing a strong hint for future forensic reverse-engineering.

Key Results

  • ResNet-18 Performance: The deep learning approach significantly outperformed the analytical method, achieving a top classification accuracy of 92.75% with a loss value of 0.23.
  • Analytical Performance: Among the traditional classifiers trained on DCT $\beta$ coefficients, the Random Forest model performed best, reaching 81% accuracy. Other classifiers like k-NN and linear SVM hovered around 73–75% accuracy.
Training accuracy and loss curves for ResNet-18

Limitations and Future Work

  • Architecture Specificity: The current study evaluates its methods exclusively on images generated by StarGAN-V2. It remains to be seen how well these forensic traces generalize across different generative architectures or when multiple different networks are mixed sequentially.
  • Manipulation Types: The focus was strictly limited to the style-transfer category of attribute manipulation. The authors note that future work must expand these ballistics techniques to other common deepfake categories, such as face reenactment (expression swap) and face swapping, as well as extending the analysis to audio and video deepfakes.

Summary updated

Abstract

Most recent style-transfer techniques based on generative architectures are able to obtain synthetic multimedia contents, or commonly called deepfakes, with almost no artifacts. Researchers already demonstrated that synthetic images contain patterns that can determine not only if it is a deepfake but also the generative architecture employed to create the image data itself. These traces can be exploited to study problems that have never been addressed in the context of deepfakes. To this aim, in this paper a first approach to investigate the image ballistics on deepfake images subject to style-transfer manipulations is proposed. Specifically, this paper describes a study on detecting how many times a digital image has been processed by a generative architecture for style transfer. Moreover, in order to address and study accurately forensic ballistics on deepfake images, some mathematical properties of style-transfer operations were investigated.
BibTeX
@misc{guarnera2022deepfake,
  author = {Luca Guarnera and Oliver Giudice and Sebastiano Battiato},
  title = {{Deepfake Style Transfer Mixture: a First Forensic Ballistics Study on Synthetic Images}},
  year = {2022},
  eprint = {2203.09928},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.09928},
}
Figure