MUN: Image Forgery Localization Based on M³ Encoder and UN Decoder

Yaqi Liu, Shuhuan Chen, Haichao Shi, Xiao-Yu Zhang, Song Xiao, Qiang CaiAAAI 2025 · DOI

TL;DR

This paper introduces MUN, a novel deep learning framework for image forgery localization that effectively combines RGB and noise-based features using a multi-scale query module. MUN achieves state-of-the-art performance by leveraging a dual-stream encoder and a UN decoder architecture, along with a novel IoUDCE loss function and a data augmentation technique (DNA) to improve generalization.

What Problem Does This Paper Address?

The proliferation of sophisticated image editing tools has made it increasingly easy to create realistic forged images, posing significant threats to public security and trust. While various methods exist for detecting specific types of forgeries (e.g., splicing, copy-move, removal), a major challenge is to develop a unified approach that can accurately localize any type of forgery within an image without prior knowledge of the tampering method. Existing methods often struggle with generalization across different forgery types and datasets, and can be sensitive to post-processing operations. This paper aims to address this by proposing a robust and versatile image forgery localization network.

Methods

The core of MUN is its novel M³ encoder, which employs a dual-stream architecture. One stream processes the standard RGB image, while the other processes a Noiseprint++ map. Noiseprint++ is a powerful tool that captures camera-specific noise patterns, acting as a "camera model fingerprint" and providing valuable low-level clues about image manipulation. The paper investigates the optimal deployment of Noiseprint++ and finds that applying it before resizing the image yields the best results.

The M³ encoder uses ConvNeXt V2 as its backbone for feature extraction in both streams. A key innovation is the Multi-scale Max-pooling Query (MMQ) module. This module integrates features from the RGB and Noiseprint++ branches by using multi-scale max-pooling on RGB features to query and correlate with the noise features. This allows the network to capture complex relationships between the two domains.

The encoder's hierarchical features are then fed into a UN decoder. This decoder features parallel U and N branches designed to extract both low-level (high spatial detail) and high-level (semantic) features from top-down and bottom-up directions, respectively. Concatenating the outputs of these branches allows for a more comprehensive reconstruction of the predicted forgery mask.

To further enhance performance, the authors introduce two novel components:

  1. IoU-recalibrated Dynamic Cross-Entropy (IoUDCE) loss: This loss function dynamically adjusts the weights of positive and negative samples based on the Intersection over Union (IoU) of the predicted mask. This helps the network focus more on challenging or misclassified regions.
  2. Deviation Noise Augmentation (DNA): This data augmentation technique aims to narrow the distribution gap between training images and universal images by adjusting RGB channel means. This improves the model's generalization ability to unseen data.

Datasets

The paper trains and evaluates MUN on several datasets:

  • Synthesized Dataset: Used for training and ablation studies. This dataset is built upon CASIA v2.0 and ADE20k and includes splicing, copy-move, and removal forgeries.
  • Evaluation Datasets:
  • NIST16
    • CASIA v1.0
    • IMD2020
    • CocoGlide (an AI-generated dataset)
    • Wild

Key Insights and Ideas

  • Dual-stream processing is crucial: Combining RGB information with low-level noise patterns (Noiseprint++) significantly improves forgery localization accuracy.
  • Optimal Noiseprint++ deployment matters: The order of operations (resizing vs. Noiseprint++ application) has a notable impact on performance, with "resize after Noiseprint++" being the most effective.
  • Multi-scale feature correlation is key: The MMQ module effectively bridges the gap between RGB and noise features by querying and integrating them at multiple scales.
  • Hierarchical UN decoder captures diverse features: The parallel U and N branches in the decoder are essential for extracting both fine-grained and abstract forgery characteristics.
  • Dynamic loss functions improve focus: IoUDCE loss helps the model adaptively learn from difficult examples by re-weighting samples based on prediction accuracy.
  • Data augmentation for generalization: DNA is a simple yet effective method to improve the model's ability to generalize to unseen image distributions.

Key Results

MUN demonstrates state-of-the-art performance across multiple benchmark datasets. For instance, on the Synthesized Dataset, MUN achieves an F1 score of 0.9533, IoU of 0.9170, Accuracy of 0.9945, and AUC of 0.9989. The paper also shows that MUN performs well on AI-generated images (CocoGlide), indicating its versatility. The DNA augmentation further boosts performance on most datasets, with MUN* (trained with DNA) outperforming MUN.

Limitations and Future Work

  • Noiseprint++ memory constraints: For ultra-high-resolution images (like those in NIST16), processing Noiseprint++ directly can be memory-intensive. The paper addresses this by patching, but it suggests a potential area for optimization.
  • Generalization to diverse noise distributions: While DNA helps, the paper notes that further exploration into noise distributions that best encompass various image distributions could be beneficial.
  • Integrating additional clues: The authors suggest that future work could involve integrating more diverse clues beyond RGB and Noiseprint++ to further enhance performance.
  • Frequency domain information: Exploring the relationship with frequency domain information is also proposed as a future research direction.

Summary updated

BibTeX
@inproceedings{liu2025mun,
  author = {Yaqi Liu and Shuhuan Chen and Haichao Shi and Xiao-Yu Zhang and Song Xiao and Qiang Cai},
  title = {{MUN: Image Forgery Localization Based on M³ Encoder and UN Decoder}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i6.32606},
}
Figure