ChatGPT’s ability to generate realistic experimental images poses a new challenge to academic integrity

Lingxuan Zhu, Yancheng Lai, Weiming Mou, Haoran Zhang, Anqi Lin, Chang Qi, Tao Yang, Liling Xu, Jian Zhang, Peng LuoJournal of Hematology & Oncology 2024 · DOI · OpenAlex

TL;DR

While early concerns about AI in academia focused on "paper mills" generating fake text, a new frontier of fraud has emerged: the generation of realistic synthetic experimental images. This paper demonstrates that ChatGPT (via DALL-E 3) can now produce plausible biomedical results—like Western blots and blood smears—with almost no technical barrier, necessitating urgent updates to how we verify scientific evidence.

What Problem Does This Paper Address?

In biomedical research, images are often the primary evidence used to support a conclusion. Whether it’s the intensity of a protein band in a Western blot or the morphology of cells in a tissue stain, these visuals are the "ground truth" of a study.

Historically, image manipulation involved manual "photoshopping" (cloning, splicing, or rotating), which left detectable digital traces. More recently, sophisticated actors used Generative Adversarial Networks (GANs) to create entirely fake images. However, GANs require significant technical expertise to train and deploy. The authors argue that the integration of DALL-E 3 into ChatGPT has "democratized" this threat. Now, anyone with a basic subscription and a simple text prompt can generate high-quality, fabricated experimental data, posing a systemic risk to the integrity of the scientific record.

Methods

The authors conducted a series of "simple tests" to probe the limits of ChatGPT’s image generation capabilities. Rather than building a new model, they acted as "adversarial users," providing ChatGPT with text prompts to generate common types of biomedical experimental results.

The goal was to determine if the current iteration of the tool could produce images that were visually indistinguishable from real data to the casual or even semi-experienced eye. They specifically targeted five common modalities:

  1. Blood smears
  2. Immunofluorescence staining
  3. Hematoxylin and eosin (H&E) staining
  4. Immunohistochemistry
  5. Western Blot (WB) images

Datasets

As this is a correspondence piece focused on a proof-of-concept, the authors did not use or introduce a large-scale training dataset. Instead, they generated a representative set of synthetic images to serve as a qualitative benchmark. They reference the "Supplementary Material" for the specific prompts used to generate these images, though the prompts themselves are not detailed in the main text.

Key Insights and Ideas

  • The "Barrier-Free" Threat: The most significant insight is the shift in the "barrier to entry." Fraud that previously required a data scientist can now be performed by anyone. This "nearly barrier-free" access is what makes LLM-integrated image generation more dangerous than previous GAN-based methods.
  • Hybrid Fraud: The authors highlight a subtle but dangerous use case: using AI to partially modify real images. For example, a researcher could use AI to selectively enhance or weaken the intensity of a specific band in a real Western blot to "prove" a desired result. These hybrid images (part real, part AI) are significantly harder to detect than fully synthetic ones.
  • Visual Plausibility vs. Perfection: While the images aren't perfect, the authors note they are "striking" and "very close" to real results. They are particularly effective at generating blood smears and immunofluorescence, which could easily fool reviewers or researchers who are not specialists in those specific techniques.

Key Results

The paper presents a figure (Fig. 1) showcasing the successful generation of five distinct types of experimental data:

  • Blood Smears & Immunofluorescence: These were identified as the most realistic and potentially deceptive outputs.
  • Western Blots: The AI successfully generated the characteristic "bands" used to measure protein concentration.
  • Histology (H&E and IHC): The model produced plausible tissue structures and staining patterns.

While the authors acknowledge that ChatGPT-generated images are currently less realistic than those produced by specialized GANs, they emphasize that the gap is closing rapidly due to the fast pace of LLM development.

Limitations and Future Work

Limitations: * Current Realism: The authors admit that a highly trained eye can still distinguish current AI-generated images from real experimental data in many cases. * Scope: The study is a qualitative "red-teaming" exercise rather than a quantitative analysis of detection rates among human experts.

Future Work & Recommendations: The authors propose a three-pronged approach to mitigate these risks:

  1. Restriction: AI providers should implement guardrails to prevent the generation of specific "experimental-style" images.
  2. Detection Tools: There is an urgent need for "AI-image detectors," similar to the tools currently used to flag AI-generated text (e.g., GPTZero).
  3. Watermarking: AI companies should adopt "invisible watermarks"—metadata or steganographic markers that are invisible to humans but easily identified by software—to ensure the provenance of generated images can always be traced.

Summary updated

Abstract

The rapid advancements in large language models (LLMs) such as ChatGPT have raised concerns about their potential impact on academic integrity. While initial concerns focused on ChatGPT's writing capabilities, recent updates have integrated DALL-E 3's image generation features, extending the risks to visual evidence in biomedical research. Our tests revealed ChatGPT's nearly barrier-free image generation feature can be used to generate experimental result images, such as blood smears, Western Blot, immunofluorescence and so on. Although the current ability of ChatGPT to generate experimental images is limited, the risk of misuse is evident. This development underscores the need for immediate action. We suggest that AI providers restrict the generation of experimental image, develop tools to detect AI-generated images, and consider adding "invisible watermarks" to the generated images. By implementing these measures, we can better ensure the responsible use of AI technology in academic research and maintain the integrity of scientific evidence.
BibTeX
@article{zhu2024chatgpts,
  author = {Lingxuan Zhu and Yancheng Lai and Weiming Mou and Haoran Zhang and Anqi Lin and Chang Qi and Tao Yang and Liling Xu and Jian Zhang and Peng Luo},
  title = {{ChatGPT’s ability to generate realistic experimental images poses a new challenge to academic integrity}},
  journal = {Journal of Hematology \& Oncology},
  year = {2024},
  doi = {10.1186/s13045-024-01543-8},
}

Loading related content...

Figure