Abstract
NeXT-IMDL is a massive new benchmark designed to expose the "benchmark illusion" in image manipulation detection, where models appear robust on paper but fail in real-world scenarios. By testing across 32 different editing tools and four critical axes—models, types, semantics, and granularity—the benchmark reveals that current state-of-the-art detectors are surprisingly fragile when faced with unseen AI-generated content.
Introduction
The rise of user-friendly generative AI (AIGC) has made high-quality image manipulation accessible to everyone, creating a major challenge for digital forensics. While many Image Manipulation Detection and Localization (IMDL) models report high scores, they often suffer from a "benchmark illusion." They perform well on the specific datasets they were trained on but crumble when encountering new editing techniques or different types of content. Existing benchmarks are often too narrow, focusing on a few manipulation types or models. This paper addresses the need for a truly generalizable and diagnostic evaluation framework to guide the development of more robust, next-generation forensics tools.
Methods and Framework
The authors propose a comprehensive framework that decouples AIGC-based manipulations into four fundamental axes:
- Editing Models: Tests if detectors overfit to the "fingerprints" of specific architectures (e.g., Stable Diffusion vs. FLUX).
- Manipulation Types: Evaluates conceptual understanding by testing across removal, replacement, and addition tasks.
- Content Semantics: Probes for "shortcut learning" where models might just flag specific objects (like humans or animals) rather than actual artifacts.
- Forgery Granularity: Assesses the ability to detect edits of varying sizes, from tiny localized patches to large-scale changes.
To build the NeXT-IMDL dataset, a sophisticated pipeline was used involving diverse proposal generation via four Multimodal Large Language Models (MLLMs) and precise localization using LangSAM.
The NeXT-IMDL Dataset
The primary contribution is the NeXT-IMDL benchmark, which includes 558,269 samples generated using 32 distinct academic and commercial editing tools. These include state-of-the-art open-source models like Stable Diffusion variants and FLUX, alongside commercial tools such as Photoshop and Meitu. Source images were curated from established datasets like MS COCO, Flickr30k, and OpenImages V7. NeXT-IMDL offers superior diversity in terms of model types, conditions (mask, text, reference image), and manipulation types compared to previous benchmarks.
Key Insights and Results
The authors benchmarked 11 representative models using five rigorous cross-dimension protocols, leading to several critical findings:
- Performance Collapse: While models achieved satisfactory results in their original "in-domain" settings, they experienced dramatic performance degradation in cross-setting evaluations.
- The "Universal Donor" Effect: Models trained exclusively on object removal tasks show surprisingly good zero-shot performance on replacement and addition tasks, as removal forces the learning of intrinsic "filling" artifacts.
- Foundation Model Advantage: MaskCLIP, which synergizes CLIP and MAE, consistently outperformed specialized artifact extractors, suggesting a shift toward leveraging large-scale pre-trained models for forensics.
- Semantic and Granularity Brittleness: Most detectors are heavily biased by the semantics and size of the edits seen during training, leading to significant performance drops when these factors change.
Conclusion and Future Work
The study acknowledges that even the best current models are far from being "universal" detectors. Future research should focus on exploring other foundation model combinations (e.g., integrating SAM or Diffusion U-Nets), mitigating semantic bias through adversarial training strategies, and adapting forensic tools to keep pace with the rapid evolution of generative models.
Summary updated