Learning to identify image manipulations in scientific publications

Ghazal Mazaheri, Kevin Urrutia Avila, Amit K. Roy-ChowdhuryarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel framework for automatically detecting duplicated images in scientific publications, a common form of scientific misconduct. By combining advanced image processing techniques with a deep learning Siamese network, their method achieves a significant improvement in accuracy over existing approaches, highlighting the critical role of pre-processing in this task.

What Problem Does This Paper Address?

Scientific publications rely heavily on images to convey experimental results and support findings. However, the integrity of these publications can be compromised by various forms of misconduct, including fabrication, falsification, and plagiarism. Image duplication, where parts of an image are copied and reused within the same paper to represent different experiments, is a particularly prevalent issue. This not only undermines the credibility of research but can also have serious consequences in fields like medicine. Current methods for detecting such manipulations often struggle with the diverse formats and inherent noise in scientific images, necessitating a more robust and accurate approach.

Methods

The proposed framework consists of three main stages:

  1. Figure Extraction: This initial step involves extracting relevant figures from scientific papers. The authors leverage Digital Object Identifiers (DOIs) and figure numbers to precisely locate and retrieve these images, preparing them for further processing.
  2. Region of Interest (ROI) Generation: Scientific figures, especially western blots (a common target for duplication), often contain extraneous information like labels and text that can interfere with detection. This stage focuses on removing this noise and isolating the critical regions where duplications are most likely to occur. This is achieved by identifying rectangular blocks (characteristic of western blots) and masking out everything else.
  3. Duplication Detection using Siamese Network: The core of the detection mechanism is a Siamese neural network. This network, characterized by two identical sub-networks sharing weights, is trained to learn the similarity between pairs of images. By feeding segmented image pairs (both duplicated and unduplicated) into this network, it learns to classify whether a given pair represents a copied section or not. The network employs convolutional layers, ReLU activation, and max-pooling, followed by a fully connected layer for binary classification.

Datasets

The primary dataset used in this work is derived from the dataset provided by Bik [1]. This dataset contains scientific papers with identified instances of inappropriate image duplications, often marked with bounding boxes. The authors adapted this dataset by:

  • Extracting figures based on DOIs and figure numbers.
  • Generating ROIs to focus on relevant image sections.
  • Segmenting these ROIs to create pairs for the Siamese network.
  • Carefully handling duplicated and unduplicated images, removing bounding boxes from duplicated regions to create clean training data.

Key Insights and Ideas

  • Pre-processing is Paramount: The paper strongly emphasizes that effective image pre-processing (figure extraction and ROI generation) is crucial for achieving high accuracy in duplication detection. Simply feeding raw figures into detection models yields significantly lower results.
  • Western Blots as a Focus: The authors observed that image duplication is most frequent in western blot images, which often have a distinct rectangular block structure. This observation guided their ROI generation strategy.
  • Siamese Networks for Similarity: Siamese networks are well-suited for tasks involving similarity comparison, making them an excellent choice for detecting copied image segments, especially when training data might be limited.
  • Improving Accuracy by ~13%: The proposed method demonstrates a substantial improvement in detection accuracy compared to existing state-of-the-art methods, particularly when the latter are not equipped with similar pre-processing steps.

Key Results

The proposed framework achieved a detection accuracy of 90.86%. This represents approximately a 13% improvement compared to other manipulation detection methods when evaluated on the same dataset. The paper also highlights that using ROI-processed figures as input for established methods like XceptionNet and MesoNet significantly boosted their performance over using raw figures, underscoring the effectiveness of their pre-processing pipeline.

Limitations and Future Work

  • Focus on Duplication: The current framework specifically targets image duplication. While this is a common issue, other forms of image manipulation (e.g., splicing, retouching) are not addressed.
  • Western Blot Bias: The ROI generation strategy is heavily influenced by the structure of western blots. While effective for this type of image, its applicability to other figure types (charts, graphs) might require further adaptation.
  • Future Directions: The authors suggest that their framework can be extended to detect other types of image manipulations beyond simple duplication. Further research could also explore more generalized ROI extraction techniques to handle a wider variety of scientific figure formats.

Summary updated

Abstract

Adherence to scientific community standards ensures objectivity, clarity, reproducibility, and helps prevent bias, fabrication, falsification, and plagiarism. To help scientific integrity officers and journal/publisher reviewers monitor if researchers stick with these standards, it is important to have a solid procedure to detect duplication as one of the most frequent types of manipulation in scientific papers. Images in scientific papers are used to support the experimental description and the discussion of the findings. Therefore, in this work we focus on detecting the duplications in images as one of the most important parts of a scientific paper. We propose a framework that combines image processing and deep learning methods to classify images in the articles as duplicated or unduplicated ones. We show that our method leads to a 90% accuracy rate of detecting duplicated images, a ~ 13% improvement in detection accuracy in comparison to other manipulation detection methods. We also show how effective the pre-processing steps are by comparing our method to other state-of-art manipulation detectors which lack these steps.
BibTeX
@misc{mazaheri2021learning,
  author = {Ghazal Mazaheri and Kevin Urrutia Avila and Amit K. Roy-Chowdhury},
  title = {{Learning to identify image manipulations in scientific publications}},
  year = {2021},
  eprint = {2102.01874},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2102.01874},
}
Figure