Abstract: In recent years, visual forgery has reached a level of sophistication that humans cannot identify fraud, which poses a significant threat to information security. A wide range of malicious applications have emerged, such as fake news, defamation or blackmailing of celebrities, impersonation of politicians in political warfare, and the spreading of rumours to attract views. As a result, a rich body of visual forensic techniques has been proposed in an attempt to stop this dangerous trend. In this paper, we present a benchmark that provides in-depth insights into visual forgery and visual forensics, using a comprehensive and empirical approach. More specifically, we develop an independent framework that integrates state-of-the-arts counterfeit generators and detectors, and measure the performance of these techniques using various criteria. We also perform an exhaustive analysis of the benchmarking results, to determine the characteristics of the methods that serve as a comparative reference in this never-ending war between measures and countermeasures.
TL;DR
This paper introduces the first independent "dual benchmark" designed to evaluate the ongoing arms race between facial forgery generators and forensic detectors. By testing eight state-of-the-art detection methods against eight diverse forgery techniques across a massive new dataset (DBD), the authors provide a comprehensive survival guide for identifying deepfakes under real-world adversarial conditions like heavy compression and low resolution.
The Need for a Unified Framework
The "war" between deepfake creators and forensic investigators is a classic cat-and-mouse game. As soon as a new detection method is published, forgery techniques evolve to bypass it. Historically, benchmarking these tools has been fragmented: researchers often test their detectors against a limited range of forgeries or on small, high-quality datasets that don't reflect the messy reality of the internet.
The authors address this lack of "common ground" by creating a unified framework. They evaluate two sides of the coin:
- Visual Forgery Paradigms: Categorized into Graphics-based (using handcrafted features like 3D landmarks) and Feature-based (using Deep Neural Networks like GANs and Autoencoders).
- Visual Forensics Paradigms: Pitting traditional Computer Vision detectors (looking for frequency anomalies or head-pose inconsistencies) against Deep Learning detectors (like XceptionNet and Capsule networks).
The Dual-benchmarking Dataset (DBD)
To support this rigorous evaluation, the authors introduced a massive new dataset called the Dual-benchmarking dataset (DBD), which significantly outscales previous benchmarks in both volume and diversity.
- Scale: The DBD includes 100,000 real images and 1 million synthetic images, alongside 19,000 real videos and over 21,000 synthetic videos.
- Diversity: It covers three primary forgery types: identity swaps (Id Swap), attribute swaps (Att Swap), and attribute manipulation (Att mani), such as changing hair or skin color.
- External Validation: The framework also incorporates established sets like Celeb-DF, FaceForensics++ (FF++), and DeepFake-in-the-wild to ensure broad applicability.
Key Insights and Analysis
Beyond simple accuracy, the study uses an attention-based layer to visualize where models are looking, revealing several critical insights:
- The Generalization Gap: Most detectors are "brittle." If a model is trained to find FaceSwap-2D, it often fails when encountering a GAN-based forgery. Deep learning models struggle to generalize to "unseen" forgery techniques unless the underlying tech is similar.
- The "Fingerprint" Advantage: While XceptionNet is the most accurate in perfect conditions, GAN-fingerprint is the most robust. It looks for the specific "digital DNA" left by generative models, allowing it to work even when images are tiny or noisy.
- Adversarial Sensitivity: Deep learning detectors are highly sensitive to image quality. A simple drop in resolution or an increase in JPEG compression can tank the accuracy of a state-of-the-art detector from 95% to 50%.
Performance Results
- Top Performer: XceptionNet achieved the best overall performance, often reaching >90% accuracy across multiple datasets in ideal conditions.
- Speed: Deep learning wins on efficiency. XceptionNet and Capsule processed 10,000 images in roughly 23–26 seconds, while traditional computer vision methods (like HPBD) took up to 9,100 seconds for the same task.
- Robustness Champion: When resolution was dropped to a measly 16x16 pixels, most detectors failed, but GAN-fingerprint maintained an accuracy above 0.6.
Conclusion
The authors acknowledge that even the best forensic tools still struggle with "unseen" forgeries. While the benchmark is comprehensive for images, more work is needed specifically on video-specific forgeries, where temporal consistency (flickering between frames) provides a different set of clues. The goal is to keep the DBD benchmark "living" by integrating new forgery methods as they emerge in this never-ending war.
Summary updated