Exposing image splicing traces in scientific publications via uncertainty-guided refinement

Xun Lin, Wenzhong Tang, Haoran Wang, Yizhong Liu, Yakun Ju, Shuai Wang, Zitong YuPatterns 2024 · DOI

TL;DR

This paper introduces an Uncertainty-guided Refinement Network (URN) to detect image splicing in scientific publications, a challenging task due to disruptive factors and limited data. The URN leverages uncertainty estimation to improve robustness and accuracy, outperforming existing methods and introducing a new, diverse dataset (SciSp) for this critical problem.

What Problem Does This Paper Address?

Scientific publications are increasingly scrutinized for image manipulation, with splicing being a common and challenging form of forgery. Unlike image duplication or synthesis, splicing involves combining parts of different images, often resulting in localized tampering that is hard to detect without reference images. Furthermore, scientific images themselves often contain "disruptive factors" like compression artifacts, abnormal patterns, and noise, which can mimic splicing traces and mislead detection algorithms. Existing methods, primarily developed for natural images, struggle with these challenges, leading to a need for specialized approaches and datasets for scientific image splicing detection.

Methods

The core of this work is the Uncertainty-guided Refinement Network (URN), a two-stage deep learning framework.

  • Stage 1: This stage uses a CNN encoder-decoder architecture with Monte Carlo Dropout (MCD) to generate coarse predictions of spliced regions and, crucially, estimate pixel-level uncertainty maps. This uncertainty estimation helps identify areas prone to misclassification due to disruptive factors.
  • Stage 2: This stage refines the coarse predictions by integrating the uncertainty information. It employs two novel modules:
  • Uncertainty-Guided Graph Convolution (UGGC): This module uses a local directed graph to control information flow between regions. It restricts unreliable information from uncertain regions while guiding them to learn from confident neighbors, enhancing robustness against disruptive factors.
  • Uncertainty-Enhanced Manipulation Attention (UEMA): This module focuses attention on uncertain regions during the decoding phase, helping to distinguish spliced areas from pristine ones by leveraging both uncertainty maps and robust features from UGGC.

Datasets

The authors introduce a new dataset called SciSp for scientific image splicing detection. SciSp is designed to be more diverse and challenging than existing datasets. It consists of two subsets:

  • SciSp-C: Collected from public comments on the PubPeer platform, containing 290 images with raw annotations from academic peers.
  • SciSp-H: Manually crafted spliced images (1,000 images) using five different splicing approaches and post-processing techniques to minimize visible traces, making it a more challenging benchmark.

The paper also evaluates their method on existing datasets: Biofors and RSIID.

Key Insights and Ideas

  • Uncertainty is Key: The paper strongly advocates for leveraging uncertainty estimation not just to identify potential forgeries but also to guide the refinement process, making the detection more robust to image noise and artifacts.
  • Disruptive Factors are a Major Hurdle: The authors highlight that artifacts and abnormal patterns inherent in scientific images are significant challenges that existing methods fail to address effectively.
  • Scientific vs. Natural Images: There's a clear distinction in the challenges posed by scientific images compared to natural images, necessitating specialized approaches rather than direct application of existing methods.
  • Dataset Diversity Matters: The creation of SciSp, with its diverse sources and splicing techniques, is crucial for developing models that generalize well to real-world scenarios.

Key Results

  • The proposed URN significantly outperforms nine state-of-the-art image manipulation detection methods designed for natural images across multiple benchmark datasets (Biofors, RSIID, SciSp-C, SciSp-H).
  • URN achieves the highest average F1-scores and MCC values for pixel-level detection and competitive AUC and Accuracy scores for image-level detection.
  • The method demonstrates strong robustness against various post-processing techniques, including advanced AI-based inpainting methods like LaMa and Stable Diffusion v2.
  • URN shows good generalisability against domain shifts, performing well in cross-dataset testing.

Limitations and Future Work

  • Domain Generalization: While URN shows good generalisability, cross-dataset results are not as inspiring as intra-dataset results, indicating room for improvement in adapting to different types of scientific images or even images from other fields.
  • Imbalanced Data: There's a significant imbalance between pristine and spliced images, and future work could explore how to effectively leverage the vast number of pristine images.
  • Accuracy: The current accuracy of the method, while superior, remains inadequate for fully automated validation, necessitating expert review for real-world applications.
  • Adversarial Manipulation: The existence of such methods might push malicious researchers to develop more sophisticated image manipulation techniques.

Summary updated

BibTeX
@article{lin2024exposing,
  author = {Xun Lin and Wenzhong Tang and Haoran Wang and Yizhong Liu and Yakun Ju and Shuai Wang and Zitong Yu},
  title = {{Exposing image splicing traces in scientific publications via uncertainty-guided refinement}},
  journal = {Patterns},
  year = {2024},
  doi = {10.1016/j.patter.2024.101038},
}
Figure