NeXT-IMDL: Build Benchmark for NeXT-Generation Image Manipulation Detection & Localization

Yifei Li, Haoyuan He, Yu Zheng, Bingyao Yu, Wenzhao Zheng, Lei Chen, Jie Zhou, Jiwen LuSubmission to ICLR 2026 · Withdrawn

TL;DR

Current AI-generated image manipulation detectors suffer from a "benchmark illusion": they achieve high scores on standard benchmarks but collapse when exposed to unseen editing tools, semantic concepts, or edit sizes. To diagnose this fragility, the authors introduce NeXT-IMDL, a diagnostic benchmark of over 550,000 manipulated images spanning 32 generative editing tools. Through five cross-dimension protocols, the study reveals systemic generalization failures in existing detectors and identifies why training on low-semantic tasks (like object removal) yields the most robust forgery features.

What Problem Does This Paper Address?

Modern generative image editing tools (from Stable Diffusion and FLUX to Photoshop Generative Fill and multimodal LLMs) allow anyone to manipulate images with photorealistic quality. In response, researchers have built numerous Image Manipulation Detection and Localization (IMDL) algorithms.

However, existing evaluations typically rely on simple cross-dataset testing or narrow datasets focused on a single editing technique (e.g., mask-based diffusion inpainting). These tests create a misleading impression of progress. In reality, detectors often overfit to specific generator "fingerprints," particular object categories, or specific edit scales. When evaluated on real-world, out-of-distribution edits, model performance drops drastically.

Methods

To systematically diagnose detector vulnerabilities, the paper decouples generative manipulation into four fundamental axes:

  1. Editing Models: Testing sensitivity to generator architectures (e.g., GANs, diffusion variants, commercial tools).
  2. Manipulation Types: Assessing whether detectors learn general forgery traces across tasks (Removal, Replacement, Addition, and Null-Text editing).
  3. Content Semantics: Checking for "shortcut learning," where models flag images based on semantic content (e.g., humans vs. animals vs. objects) rather than manipulation artifacts.
  4. Forgery Granularity: Probing multi-scale localization across varying edit sizes (small, medium, large).

Using these axes, the authors formulate five cross-dimension evaluation protocols:

  • Cross-EM (Cross-Edit-Models): Train on one subset of generators, test on disjoint generators.
  • Cross-ET (Cross-Edit-Types): Train on one edit type (e.g., removal) and evaluate zero-shot on others (e.g., addition).
  • Cross-SL (Cross-Semantic-Labels): Train on specific object classes and test on unseen categories.
  • Cross-EG (Cross-Edit-Granularity): Train and test across disjoint edit sizes.
  • RealWorld-IMDL: Measure the lab-to-wild generalization gap by training strictly on academic models and evaluating on commercial tools (e.g., Adobe Photoshop, Doubao).

The authors benchmarked 11 representative models: 6 pixel-level IMDL localization methods (e.g., MaskCLIP, TruFor, MVSS-Net, PSCC-Net, IML-ViT, Mesorch) and 5 image-level binary AIGC detectors (e.g., UniFD, FreqNet, NPR, AIDE, FIRE).

Datasets

The authors construct NeXT-IMDL, comprising 558,269 manipulated images paired with corresponding pristine images and pixel-level ground-truth masks:

  • Source Images: Sampled from Flickr30k, MS COCO, and OpenImages V7.
  • Instruction & Mask Generation: Four VLM families (InternVL, Qwen-VL, GPT-4o, and Gemini) proposed editing instructions, while LangSAM generated localized masks.
  • Generative Engines: 32 academic, open-source, and commercial editing tools, including SDXL, FLUX, Inpaint-Anything, CLIPAway, Adobe Photoshop, GPT-Image-1, and Gemini-2.0-flash-Image.
  • Quality Filtering: Edits passed through automated metric filtering (SSIM, CLIP similarity) and manual verification.

Key Insights and Ideas

  • The "Universal Donor" Effect of Object Removal: Models trained exclusively on object removal transferred surprisingly well to object replacement and addition in a zero-shot manner (e.g., MaskCLIP achieved 0.385 IoU on replacement after training only on removal). Conversely, models trained on object replacement or addition performed poorly on removal. Because object removal strips away the newly introduced object semantics, it forces the network to learn the fundamental artifacts of generative image synthesis rather than semantic shortcuts.
  • The "Signal Drowning" Effect: Global binary AIGC detectors systematically failed on localized generative edits, frequently collapsing to near-random guessing (~0.50 accuracy). When only a small patch is manipulated, the authentic portions of the image dominate the global feature representation, completely drowning out the local forgery signal.
  • Asymmetric Semantic and Granularity Brittleness: Detectors exhibit heavy biases toward specific object categories and sizes. For example, MaskCLIP experienced significant performance drops when trained only on human subjects compared to non-human objects. Across granularity, training on large manipulated regions (Area-3) yielded the best transfer to smaller regions, whereas models trained on small regions failed to generalize upward.
  • The Dominance of Synergized Foundation Models: MaskCLIP consistently outperformed specialized artifact extractors across all protocols. The authors attribute this to pairing CLIP’s broad, robust semantic pre-training with MAE’s fine-grained, patch-level reconstruction representations.

Key Results

  • Pixel-Level Localization (IoU / F1):
  • Under Protocol-1 (Cross-Edit-Models for Removal), MaskCLIP achieved the highest average localization score (0.288 IoU / 0.354 F1), while traditional detectors struggled (e.g., PSCC-Net scored 0.092 IoU / 0.126 F1, and MVSS-Net reached 0.126 IoU / 0.155 F1).
  • Under Protocol-4 (Granularity), training on small edits (Area-1) caused severe localization drops on large edits (Area-3), yielding average IoUs as low as 0.002 (TruFor) and 0.012 (MaskCLIP). Conversely, training on Area-3 allowed MaskCLIP to reach 0.337 mean IoU.
  • Image-Level Binary Detection:
  • Under Protocol-1, binary detectors showed near-chance performance when generalizing across model splits (e.g., FreqNet, NPR, and FIRE hovered between 0.49 and 0.55 on several evaluation subsets).

Limitations and Future Work

  • Residual Dataset Bias: While NeXT-IMDL incorporates 32 tools and 4 VLM families, the underlying source datasets (COCO, Flickr30k) and text-to-image base models may still encode latent geographic and cultural distribution biases.
  • Quantifying Semantic Bias: The authors note the need for formal metrics to quantify a detector's "semantic dependency" and propose exploring adversarial or contrastive losses that disentangle semantic object features from manipulation artifacts.
  • Next-Generation Backbones: While CLIP + MAE performs well, future work should explore fusing semantic extractors with generative backbones (such as diffusion U-Net feature maps) or dedicated segmentation foundations like SAM to improve boundary localization.

Summary updated

Abstract

The accessibility surge and abuse risks of user-friendly image editing models have created an urgent need for generalizable, up-to-date methods for Image Manipulation Detection and Localization (IMDL). Current IMDL research typically uses cross-dataset evaluation, where models trained on one benchmark are tested on others. However, this simplified evaluation approach conceals the fragility of existing methods when handling diverse AI-generated content, leading to misleading impressions of progress. This paper challenges this illusion by proposing NeXT-IMDL, a large-scale diagnostic benchmark designed not just to collect data, but to probe the generalization boundaries of current detectors systematically. Specifically, NeXT-IMDL categorizes AIGC-based manipulations along four fundamental axes: editing models, manipulation types, content semantics, and forgery granularity. Built upon this, NeXT-IMDL implements five rigorous cross-dimension evaluation protocols. Our extensive experiments on 11 representative models reveal a critical insight: while these models perform well in their original settings, they exhibit systemic failures and significant performance degradation when evaluated under our designed protocols that simulate real-world various generalization scenarios. By providing this diagnostic toolkit and the new findings, we aim to advance the development towards building truly robust, next-generation IMDL models.
BibTeX
@misc{li2025next,
  author = {Yifei Li and Haoyuan He and Yu Zheng and Bingyao Yu and Wenzhao Zheng and Lei Chen and Jie Zhou and Jiwen Lu},
  title = {{NeXT-IMDL: Build Benchmark for NeXT-Generation Image Manipulation Detection \& Localization}},
  year = {2025},
  eprint = {2512.23374},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.23374},
}

Loading related content...

Figure