Individualized Deepfake Detection Exploiting Traces Due to Double Neural-Network Operations

Mushfiqur Rahman, Runze Liu, Chau-Wai Wong, Huaiyu DaiarXiv 2023 · DOI · arXiv

Abstract

As deepfake technology advances, general-purpose detectors often struggle with cross-dataset generalization and consistency across different individuals. This paper introduces an individualized detection framework that exploits the "near-idempotence" of neural network-based face-swap generators. By applying a person-specific reconstruction operator and measuring the resulting "double operation" trace via a Siamese network, the proposed method achieves superior accuracy and reliability. Experimental results demonstrate an Area Under the Curve (AUC) of 0.94 and a 17% reduction in performance variance compared to state-of-the-art baselines, while maintaining robustness against sophisticated diffusion-based forgeries.

Introduction

The proliferation of high-quality deepfakes poses a significant challenge for information integrity, particularly in journalism. Most existing detection models are "generalists," trained to identify artifacts across broad populations. However, these models frequently fail when encountering unseen deepfake generation methods or specific facial characteristics of high-profile individuals.

This research argues for an "individualized" approach. By focusing on specific public figures, a detector can be conditioned on identity-specific features, allowing it to ignore natural facial variations and focus strictly on the mathematical artifacts left by forgery algorithms. The core of this approach is the exploitation of the "double stamp"—the trace left when a neural network processes an image that has already been manipulated by a similar network.

Methodology

The Principle of Near-Idempotence

In mathematics, an idempotent operation is one that can be applied multiple times without changing the result beyond the initial application. The authors observed that deepfake generators (GANs or Diffusion models) exhibit near-idempotence. An authentic image processed by a generator undergoes significant feature changes as the network adds its specific "traces." Conversely, an image that is already a deepfake undergoes only marginal changes when re-processed by the same or a similar generator, as the traces are already present.

Comparison between the proposed idempotent strategy and conventional training methods.

System Architecture

The proposed pipeline consists of three primary components:

  1. Reconstruction Operator ($R$): A person-specific autoencoder trained to mimic the behavior of common deepfake tools for a specific identity (e.g., Tom Hanks).
  2. Identity-Aware Feature Extractor: A ResNet-based backbone fine-tuned via knowledge distillation to extract features sensitive to both identity and deepfake artifacts.
  3. Siamese Network: A comparison module that evaluates the Euclidean distance between the features of an input image and its reconstructed version. A large distance indicates an authentic image (first-time "stamping"), while a small distance suggests a deepfake (second-time "stamping").

Experimental Results

To evaluate the system, the researchers curated a dataset of 32,000 images featuring 45 public figures, using Celeb-DF for training and the Cross-Age Celebrity Dataset (CACD) for testing to ensure cross-dataset robustness.

Performance metrics, system architecture overview, and t-SNE visualization of feature separation.

Performance and Reliability

The proposed method outperformed nine state-of-the-art detectors, including fine-tuned versions of Xception and EfficientNet. The system achieved a peak AUC of 0.94. Crucially for real-world deployment, the standard deviation of the AUC was reduced by 17%, indicating that the model provides consistent performance across different individuals, unlike generalist models that may exhibit high error rates for specific faces.

Robustness to Diffusion Models

While the Siamese network was primarily trained on GAN-based traces, it demonstrated remarkable effectiveness in detecting newer Diffusion-based deepfakes (such as DiffSwap). This suggests that the "double operation" trace is a fundamental characteristic of neural network-based image synthesis, regardless of the underlying architecture.

Discussion and Conclusion

The individualized strategy represents a paradigm shift in forensic verification. By moving from general-purpose detection to identity-conditioned verification, the system provides a more reliable tool for high-stakes environments like newsrooms. While the current implementation requires a pre-trained reconstruction operator for each target identity (a "closed-set" limitation), it offers a proactive defense for the most frequently targeted public figures. Future work will focus on generalizing the Siamese detector to handle a broader range of mismatched deepfake operations without requiring person-specific training for every new case.

Summary updated

Abstract

In today's digital landscape, journalists urgently require tools to verify the authenticity of facial images and videos depicting specific public figures before incorporating them into news stories. Existing deepfake detectors are not optimized for this detection task when an image is associated with a specific and identifiable individual. This study focuses on the deepfake detection of facial images of individual public figures. We propose to condition the proposed detector on the identity of an identified individual, given the advantages revealed by our theory-driven simulations. While most detectors in the literature rely on perceptible or imperceptible artifacts present in deepfake facial images, we demonstrate that the detection performance can be improved by exploiting the idempotency property of neural networks. In our approach, the training process involves double neural-network operations where we pass an authentic image through a deepfake simulating network twice. Experimental results show that the proposed method improves the area under the curve (AUC) from 0.92 to 0.94 and reduces its standard deviation by 17%. To address the need for evaluating detection performance for individual public figures, we curated and publicly released a dataset of ~32k images featuring 45 public figures, as existing deepfake datasets do not meet this criterion.
BibTeX
@misc{rahman2023individualized,
  author = {Mushfiqur Rahman and Runze Liu and Chau-Wai Wong and Huaiyu Dai},
  title = {{Individualized Deepfake Detection Exploiting Traces Due to Double Neural-Network Operations}},
  year = {2023},
  eprint = {2312.08034},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.08034},
}
Figure