AI-enabled image fraud in scientific publications

Jinjin Gu, Xinlei Wang, Chenang Li, Junhua Zhao, Weijin Fu, Gaoqi Liang, Jing Qiu

TL;DR

This paper highlights the emerging threat of AI-generated fake images being used to commit scientific fraud in publications. Advanced generative models can create realistic images that are increasingly difficult to detect, potentially undermining the integrity of scientific research and the peer-review process.

What Problem Does This Paper Address?

The scientific community relies heavily on visual evidence presented in publications. However, traditional forms of image misconduct, such as duplication and manipulation, are becoming more sophisticated with the advent of powerful AI image generation tools. These tools can create entirely new, yet convincingly realistic, scientific images that are hard to distinguish from genuine data. This poses a significant risk of deceiving editors, reviewers, and the wider scientific community, leading to the dissemination of false findings and wasted research efforts. The paper aims to raise awareness of this threat and explore potential countermeasures.

Methods

The authors discuss several ways AI generative models can be misused for image fraud:

  • Image Generation: Generative models like GANs (Generative Adversarial Networks) can learn the underlying patterns of scientific images and produce novel, high-fidelity fake images that appear scientifically consistent but are meaningless.
  • Image Regeneration/Resampling: Models like SinGAN can learn from a single image and regenerate it with different local details, making it difficult to detect through traditional duplication checks. This can be used to plagiarize existing images or create fake control groups.
  • Image Editing/Manipulation: Generative models can be used to subtly alter existing images, such as removing or adding cells or objects, making them appear to support a fabricated narrative.

The paper also evaluates the effectiveness of two state-of-the-art AI forensic tools (Wang et al., 2020 and Gragnaniello et al., 2021) in detecting these AI-generated fake images.

Datasets

The paper describes the datasets used for training and evaluation:

  • Training Datasets:
    • Cell immunostaining confocal microscope images from the Human Protein Atlas dataset.
  • Histopathological images from the breast cancer histopathological dataset (BreCaHAD).
  • Scanning micrographs of nano materials (collected from the internet).
  • Evaluation Datasets: The evaluation involved a combination of natural images, scanning micrographs, cell immunostaining images, immunohistochemistry (IHC) images, and histopathological images.

Key Insights and Ideas

  • "Seeing is Believing" is Under Threat: The increasing realism of AI-generated images challenges the fundamental reliance on visual evidence in science.
  • Subtlety of AI Forgery: Unlike traditional image manipulation, AI-generated images can be created from scratch or subtly altered, leaving fewer discernible artifacts for forensic analysis.
  • Human Experts are Vulnerable: A user study indicated that even experienced human experts struggle to reliably distinguish AI-generated scientific images from real ones, often showing less confidence in their judgments compared to natural images.
  • Limitations of Current AI Forensics: While AI tools show promise, current detection accuracy is not sufficient to fully mitigate the threat, and malicious actors can select fakes that pass detection thresholds.
  • Low Barrier to Entry: The open-source nature and accessibility of advanced generative models significantly lower the technical and financial barriers for committing image fraud.
  • Potential for "Paper Factories": The ease and scalability of AI image generation could enable systematic production of falsified research papers by entities like "paper factories."

Key Results

  • Human Visual Inspection: The user study showed that human experts had difficulty distinguishing between real and generated scientific images, with ratings often leaning towards "real" but with lower confidence.
  • AI Forensic Tool Performance:
    • Wang et al. (2020) achieved accuracy similar to human visual inspection.
  • Gragnaniello et al. (2021) performed generally better than Wang et al. but neither tool provided sufficient detection accuracy to fully counter the threat.
  • Qualitative Findings: The paper visually demonstrates how AI models can generate entirely new, yet plausible-looking, scientific images (Figure 1A), regenerate existing images with altered details (Figure 1B), and manipulate images by adding or removing elements (Figure 1C).

Limitations and Future Work

  • Limitations of Current Detection Methods: The paper acknowledges that current AI forensic tools are not yet robust enough to reliably detect all AI-generated image fraud. They are also vulnerable to malicious users who can simply select fakes that pass the detection threshold. The development of tools specialized for scientific images is also noted as an area needing improvement, as current tools are often trained on natural images.
  • Future Work:
    • Improved Management Systems: Enhancing management systems and processes within academia to address these new risks.
  • Requesting Raw Data: Authors could be asked to provide more detailed, high-resolution raw image data, as current generative models struggle with generating very large, high-fidelity images.
  • Developing Advanced Forensic Tools: Continued research and development of specialized forensic tools for advanced image generation and processing models, particularly for scientific images.
  • Considering Social Impact: When developing new image generation technologies, researchers should consider their potential social impact and explore ways to mitigate abuse, such as through encryption or steganography of generated images.
  • Enduring Cat-and-Mouse Game: The authors foresee an ongoing challenge between the development of new AI technologies and the countermeasures to prevent their abuse.

Summary updated

BibTeX
@article{gu2022ai,
  author = {Jinjin Gu and Xinlei Wang and Chenang Li and Junhua Zhao and Weijin Fu and Gaoqi Liang and Jing Qiu},
  title = {{AI-enabled image fraud in scientific publications}},
  journal = {Patterns},
  year = {2022},
  doi = {10.1016/j.patter.2022.100511},
}
Figure