Bridging Semantic Logic Gaps: A Cognition-Inspired Multimodal Boundary-Preserving Network for Image Manipulation Localization

Songlin Li, Zhiqing Guo, Yuanman Li, Zeyu Li, Yunfeng Diao, Gaobo Yang, Liejun WangarXiv 2025 · arXiv · OpenReview

Abstract

Image manipulation localization (IML) faces significant challenges as AI-driven edits become increasingly seamless, leaving fewer low-level visual artifacts. This paper introduces the Cognition Inspired Multimodal Boundary Preserving Network (CMB-Net), a framework designed to bridge the "semantic logic gap" by leveraging Large Language Models (LLMs). CMB-Net identifies logical inconsistencies—such as unnatural proportions or contextual errors—that traditional visual-only models overlook. By combining LLM-guided reasoning with a hallucination-filtering module and an invertible neural network for edge preservation, CMB-Net achieves state-of-the-art performance across multiple forensic benchmarks, demonstrating superior robustness and boundary accuracy.

Introduction

The rapid advancement of generative AI has made detecting tampered image regions—localization—a critical task in digital forensics. Traditional IML models primarily rely on low-level visual cues like noise patterns, frequency inconsistencies, or edge artifacts. However, modern high-quality edits often minimize these traces, leaving only "semantic logic gaps." These are errors in the logical structure of an image, such as a person's legs being unnaturally thin or shadows falling inconsistently. While humans easily spot these discrepancies, standard computer vision models remain largely blind to them.

Traditional paradigms in image manipulation localization focusing on visual cues and additional visual information.

Recent attempts to incorporate LLMs into IML have struggled with two main issues: "hallucinations" (where the LLM describes non-existent features) and the loss of fine-grained boundary details during multimodal feature fusion. CMB-Net addresses these by treating the LLM as a sophisticated reasoning engine while implementing rigorous filtering and high-fidelity decoding mechanisms.

Methodology

CMB-Net introduces a multimodal architecture that combines visual encoding with textual reasoning to bridge the logic gap.

The proposed CMB-Net architecture, integrating LLM-guided reasoning with visual features and an ambiguity analysis module to identify semantic inconsistencies.

The framework consists of several novel components:

  1. LLM-Guided Reasoning: The system prompts an LLM (e.g., Qwen-VL-Max) to analyze the image and describe potential tampering in natural language. This provides the "semantic logic" missing from visual features.
  2. Image-Text Central Ambiguity Module (ITCAM): To solve the LLM hallucination problem, ITCAM quantifies the ambiguity between the image and the generated text. It uses K-Nearest Neighbors (KNN) and Kullback-Leibler (KL) divergence to measure how well the text aligns with the image. If the text is irrelevant, its weight is automatically reduced.
  3. Image-Text Interaction Module (ITIM): This module facilitates fine-grained fusion using a cross-modal correlation matrix to align visual and textual features, ensuring semantic clues are mapped to the correct spatial regions.
  4. Restoration Edge Decoder (RED): Inspired by Invertible Neural Networks (INN), this decoder is designed for "lossless" boundary preservation. Because INNs can mutually generate input and output features, they prevent the "dilution" of edge information common in standard upsampling.

Experimental Setup and Results

The model was evaluated on standard benchmarks (CASIAv1/v2, NIST16, Columbia, Coverage) and real-world datasets (IMD2020, CocoGlide, ITW, Korus). A new dataset of text descriptions was created using Qwen-VL-Max and GPT-4.1 to facilitate training.

Key Results

CMB-Net outperformed eight state-of-the-art methods, including TruFor and PSCC-Net. On the NIST16 benchmark, it achieved an F1 score of 0.935 and an IoU of 0.891, significantly surpassing the next best model (MFI-Net at 0.853 F1). The model also demonstrated high resilience to routine image degradations like JPEG compression and Gaussian noise, maintaining performance across social media platforms like Facebook and Weibo.

Insights on LLM Selection

The researchers found that Qwen-VL-Max outperformed GPT-4.1. Qwen’s descriptions were more succinct and logically focused, whereas GPT-4.1’s verbosity introduced redundant details that hindered model performance.

Discussion and Conclusion

CMB-Net successfully bridges the gap between low-level visual detection and high-level semantic reasoning. Its use of ITCAM effectively manages the noise inherent in LLM outputs, while the RED module ensures the precise localization required for forensic applications.

Limitations and Future Work

Performance degradation was observed on images compressed by WeChat, likely due to aggressive compression destroying the fine-grained visual anchors needed for LLM alignment. Additionally, the inclusion of LLMs and BERT encoders results in a larger parameter count (~156M) compared to lightweight models like PSCC-Net (~3.6M). Future directions include improving robustness against extreme routine damages and exploring more parameter-efficient ways to integrate LLM reasoning.

Summary updated

Abstract

The existing image manipulation localization (IML) models mainly relies on visual cues, but ignores the semantic logical relationships between content features. In fact, the content semantics conveyed by real images often conform to human cognitive laws. However, image manipulation technology usually destroys the internal relationship between content features, thus leaving semantic clues for IML. In this paper, we propose a cognition inspired multimodal boundary preserving network (CMB-Net). Specifically, CMB-Net utilizes large language models (LLMs) to analyze manipulated regions within images and generate prompt-based textual information to compensate for the lack of semantic relationships in the visual information. Considering that the erroneous texts induced by hallucination from LLMs will damage the accuracy of IML, we propose an image-text central ambiguity module (ITCAM). It assigns weights to the text features by quantifying the ambiguity between text and image features, thereby ensuring the beneficial impact of textual information. We also propose an image-text interaction module (ITIM) that aligns visual and text features using a correlation matrix for fine-grained interaction. Finally, inspired by invertible neural networks, we propose a restoration edge decoder (RED) that mutually generates input and output features to preserve boundary information in manipulated regions without loss. Extensive experiments show that CMB-Net outperforms most existing IML models. Our code is available on https://github.com/vpsg-research/CMB-Net.
BibTeX
@misc{li2025bridging,
  author = {Songlin Li and Zhiqing Guo and Yuanman Li and Zeyu Li and Yunfeng Diao and Gaobo Yang and Liejun Wang},
  title = {{Bridging Semantic Logic Gaps: A Cognition-Inspired Multimodal Boundary-Preserving Network for Image Manipulation Localization}},
  year = {2025},
  eprint = {2508.07216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.07216},
}
Figure