Diff-ID: An Explainable Identity Difference Quantification Framework for DeepFake Detection

Chuer Yu, Xuhong Zhang, Yuxuan Duan, Senbo Yan, Zonghui Wang, Yang Xiang, Shouling Ji, Wenzhi Chen

Abstract

Despite the fact that DeepFake forgery detection algorithms have achieved impressive performance on known manipulations, they often face disastrous performance degradation when generalized to an unseen manipulation. Some recent works show improvement in generalization but rely on features fragile to image distortions such as compression. To this end, we propose Diff-ID, a concise and effective approach that explains and measures the identity loss induced by facial manipulations. When testing on an image of a specific person, Diff-ID utilizes an authentic image of that person as a reference and aligns them to the same identity-insensitive attribute feature space by applying a face-swapping generator. We then visualize the identity loss between the test and the reference image from the image differences of the aligned pairs, and design a custom metric to quantify the identity loss. The metric is then proved to be effective in distinguishing the forgery images from the real ones. Extensive experiments show that our approach achieves high detection performance on DeepFake images and state-of-the-art generalization ability to unknown forgery methods, while also being robust to image distortions.

Introduction

DeepFake detection has become a high-stakes arms race. While modern classifiers can spot known forgery techniques with near-perfect accuracy, they often crumble when faced with a new, unseen manipulation method or heavy image compression. Most detectors focus on low-level artifacts—blending boundaries, strange textures, or inconsistent head poses. However, these "forgery traces" are fragile. New synthesis methods leave fewer traces, and simple JPEG compression can wipe them out entirely.

Furthermore, these models tend to overfit. A detector trained on the Deepfakes dataset might achieve high accuracy there but drop significantly when tested on the CelebDF dataset. Diff-ID proposes a shift in strategy: instead of looking for technical "glitches," it measures the person's identity itself. Identity inconsistency is a high-level, semantic feature that is universal across face-swapping methods.

Methodology: Measuring Identity Loss

The core insight of Diff-ID is that face-swapping is inherently imperfect. Because facial identity features (like the shape of a nose) are "entangled" with attribute features (like a smile or a head tilt), the process of fusing a source identity into a target face inevitably causes a non-negligible loss of the original identity.

Visualizing identity loss: The framework highlights discrepancies in facial features like the nose when swapping identities across different targets.

To measure this, Diff-ID uses a Reference-Assisted approach:

  1. Image Reconstruction Module: Given a test image ($I_{test}$) and an authentic reference image ($I_{ref}$) of the same person, the system uses a face-swapping generator (e.g., SimSwap) to align them. It generates versions of the faces where the identity of one is mapped onto the attributes (pose, expression, etc.) of the other.
  2. Identity Difference Quantification: By aligning both images to the exact same "attribute space," the researchers isolate differences caused purely by identity. They calculate the pixel-level $L_2$ distance and the angular distance (cosine similarity) between these aligned pairs.
  3. Normalization: To ensure the metric isn't just picking up on the generator's own reconstruction errors, the identity loss is divided by the reconstruction loss.
  4. Adaptive Fine-tuning: The authors fine-tune the underlying generator with specific "identity" and "attribute" constraints, making it more sensitive to subtle identity changes while better preserving background and lighting.

Experimental Results

The researchers evaluated Diff-ID across four major benchmarks: FaceForensics++ (FF++), Google DeepFake Detection (DFD), CelebDF, and DeeperForensics-1.0 (DFo). For training (fine-tuning the generator), they used only 1,000 images from the CelebA dataset, notably without needing any fake images for training.

Generalization and Robustness

Diff-ID demonstrated remarkable generalization—the ability to work on datasets it wasn't specifically tuned for. While a standard Xception-based detector dropped to 59.4% AUC on the CelebDF dataset, Diff-ID maintained a high 91.1% AUC. On the DFD and DFo datasets, Diff-ID achieved 98.5% and 98.3% AUC respectively, outperforming most existing frame-level and video-level methods.

Furthermore, because identity is a high-level semantic feature, it survives image compression much better than low-level pixel artifacts. In tests with heavy JPEG compression (Quality Factor = 20), the performance drop was minimal (less than 2% AUC), whereas traditional artifact-based detectors often fail completely.

Conclusion

Diff-ID provides an explainable framework for DeepFake detection by producing "diff" images that visually highlight where identity is failing—often around the eyes, eyebrows, and mouth corners. While the requirement for an authentic reference image limits its use for unknown individuals, it is highly practical for protecting public figures or users on ID-verified platforms. Future work will address "face reenactment" (expression changes), where identity loss is currently less pronounced.

Summary updated

Abstract

Despite the fact that DeepFake forgery detection algorithms have achieved impressive performance on known manipulations, they often face disastrous performance degradation when generalized to an unseen manipulation. Some recent works show improvement in generalization but rely on features fragile to image distortions such as compression. To this end, we propose Diff-ID, a concise and effective approach that explains and measures the identity loss induced by facial manipulations. When testing on an image of a specific person, Diff-ID utilizes an authentic image of that person as a reference and aligns them to the same identity-insensitive attribute feature space by applying a face-swapping generator. We then visualize the identity loss between the test and the reference image from the image differences of the aligned pairs, and design a custom metric to quantify the identity loss. The metric is then proved to be effective in distinguishing the forgery images from the real ones. Extensive experiments show that our approach achieves high detection performance on DeepFake images and state-of-the-art generalization ability to unknown forgery methods, while also being robust to image distortions.
BibTeX
@article{yu2024diff,
  author = {Chuer Yu and Xuhong Zhang and Yuxuan Duan and Senbo Yan and Zonghui Wang and Yang Xiang and Shouling Ji and Wenzhi Chen},
  title = {{Diff-ID: An Explainable Identity Difference Quantification Framework for DeepFake Detection}},
  journal = {IEEE Trans. Dependable Secur. Comput.},
  year = {2024},
  doi = {10.1109/tdsc.2024.3364679},
}
Figure