A Unified Framework for Image Manipulation Localization Under Multi-Type Weak Supervision

Songlin Li, Hui Ma, Zhiqing Guo, Dan Ma, Gaobo Yang, Xin LiaoarXiv 2026 · arXiv

TL;DR

UniIML is a unified teacher-student framework that enables image manipulation localization (IML) to learn from multiple types of weak supervision—image labels, bounding boxes, scribbles, and points—simultaneously. By leveraging the Segment Anything Model (SAM) guided by a novel "Ordinal Safe Consensus" mechanism, it generates high-quality pseudo-labels that allow weakly supervised models to rival the performance of their fully supervised counterparts.

What Problem Does This Paper Address?

Identifying exactly which pixels in an image have been manipulated (splicing, copy-move, or inpainting) is critical for fighting misinformation. However, training models for this task usually requires expensive pixel-level masks, which can take over 20 minutes per image to annotate.

Weak supervision—like image-level "manipulated/authentic" labels or simple bounding boxes—is much faster to produce (seconds vs. minutes) but introduces a "silo" problem. Most existing weakly supervised methods are tailored to just one type of annotation. You can't easily train a "bounding box model" on "scribble data." This fragmentation prevents researchers from scaling up training data by mixing different available datasets.

Fig. 1. Overview of our unified weak-supervision framework. Unlike existing methods that are typically tailored to a single annotation type, our framework jointly supports image-level labels, bounding boxes, scribbles, and points.

Methods

The authors propose UniIML, a framework that treats different weak annotations as constraints within a unified representation.

  1. Unified Representation: Whether it's a point, a box, or a scribble, the input is mapped into a shared format that defines "known foreground," "known background," and "feasible spatial support."
  2. Teacher Stage (Candidate Generation): The framework uses a frozen Segment Anything Model (SAM) as a teacher. Since SAM is a general-purpose segmenter and isn't "forensics-aware," it might generate multiple plausible masks. To help it, the authors also use Qwen-VL (a vision-language model) to suggest additional prompt proposals.
  3. Ordinal Safe Consensus (OSC): This is the "secret sauce." OSC filters out SAM's candidate masks that violate the weak annotation constraints (e.g., a mask that doesn't include a known manipulated point). It then ranks the remaining masks based on three criteria: annotation consistency, SAM’s own quality scores, and cross-view stability (how much the mask changes if you flip the image).
  4. Student Stage (Distillation): A student model (SCAF) is trained using reliability-weighted distillation. It receives direct supervision from the human annotations and learns from the teacher's "soft" pseudo-labels in unlabeled regions, weighted by how confident the OSC process was about that specific mask.
Fig. 2. Overview of UniIML. Unified weak annotations, vision-language prompts, and localization priors guide a frozen SAM to generate candidate masks. OSC selects a candidate mask and assesses its selection stability. The student is trained using reliability-weighted soft pseudo-labels, direct annotation supervision, and a weighted soft Dice loss.

Datasets

The researchers evaluated UniIML across 10 different benchmarks, covering various manipulation types:

  • In-Distribution (ID): CASIAv2, Coverage, and NIST16-C.
  • Out-of-Distribution (OOD): CASIAv1, Columbia, Korus, DSO, IMD2020, CocoGlide, and ITW.

These datasets include common forgeries like copy-move and splicing, as well as more modern generative inpainting.

Key Insights and Ideas

  • SAM is "Forensics-Blind": While SAM is powerful, its internal quality scores don't correlate well with manipulation traces. The OSC mechanism is essential because it forces SAM to respect forensic constraints provided by weak labels.
  • The Power of Points: This paper is the first to systematically investigate point-level supervision for IML. While points are the "weakest" spatial cue, they are incredibly efficient to collect.
  • Stability as a Proxy for Quality: By repeating the selection process while omitting different criteria, the authors use "selection frequency" to quantify how stable (and thus likely correct) a pseudo-label is.
  • Unified > Specialized: A single model trained on a mix of weak annotations can outperform models specialized for just one type, simply because it can leverage a larger, more diverse pool of data.

Key Results

UniIML sets a new state-of-the-art for weakly supervised IML:

  • Scribble Supervision: On the CASIAv1 dataset, UniIML achieved an F1 score of 0.749, significantly outperforming the previous best (SCAF at 0.745) and even beating some fully supervised methods.
  • Bounding Box Supervision: It outperformed the specialized BoxPromptIML across almost all benchmarks.
  • Robustness: The model remained highly effective even when images were degraded by JPEG compression, Gaussian noise, or social media processing (Facebook, WeChat, etc.), showing it learns robust forensic features rather than just fragile pixel artifacts.

Limitations and Future Work

  • Point-Supervision Overfitting: The authors noted that under extremely sparse point supervision, performance can actually drop if trained for too many epochs (e.g., 70 epochs). The model begins to overfit to the noise in the generated pseudo-labels.
  • Heavy Blur: Like most forensic tools, UniIML struggles when heavy blurring wipes out the fine-grained "traces" (noise residuals, edge artifacts) left by manipulation.
  • Future Directions: The authors suggest exploring how to better integrate vision-language models to provide even more semantic context for identifying manipulated regions.

Summary updated

Abstract

Existing weakly supervised image manipulation localization (IML) methods typically rely on annotation-specific training strategies, hindering the joint use of diverse annotations within a single model. This limits the scale of usable training data and constrains further progress. We propose UniIML, a teacher-student framework that maps diverse weak annotations into a unified representation to guide the Segment Anything Model (SAM) in generating multiple candidate masks. However, SAM is designed for general-purpose segmentation and has limited sensitivity to manipulation traces; its predicted quality scores alone may therefore fail to reliably identify the candidate that best delineates the manipulated region. We introduce Ordinal Safe Consensus (OSC), which first filters candidates using trusted weak-annotation constraints encoded in the unified representation. It then ranks the remaining candidates separately by consistency with weak annotations, SAM-predicted quality scores, and cross-view stability, aggregating these rankings for an initial selection. To assess how changes in the evaluation criteria affect candidate selection, OSC repeats ranking and selection while omitting one criterion at a time. The most frequently selected candidate is adopted as the final mask, and its relative selection frequency quantifies selection stability. We construct soft pseudo-labels from the selected candidate's predicted probabilities and estimate pixel-level supervision weights based on selection stability, pixel-level agreement among candidates, and predictive certainty. The student receives direct supervision in regions defined by weak annotations and learns from soft pseudo-labels elsewhere through reliability-weighted distillation, enabling unified training across weak annotation types. Experiments show that UniIML achieves superior average performance compared with state-of-the-art IML methods.
BibTeX
@misc{li2026unified,
  author = {Songlin Li and Hui Ma and Zhiqing Guo and Dan Ma and Gaobo Yang and Xin Liao},
  title = {{A Unified Framework for Image Manipulation Localization Under Multi-Type Weak Supervision}},
  year = {2026},
  eprint = {2611.25371},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25371},
}
Figure