Medical ChatGPT – A systematic Meta-Review

Jan Egger, Malik Sallam, Gijs Luijten, Christina Gsaxner, Antonio Pepe, Jens Kleesiek, Behrus Puladi, Jianning LimedRxiv 2024 · DOI

Abstract

Since its release at the end of 2022, ChatGPT has seen a tremendous rise in attention, not only from the general public, but also from medical researchers and healthcare professionals. ChatGPT definitely changed the way we can communicate now with computers. We still remember the limitations of (voice) assistants, like Alexa or Siri, that were “overwhelmed” by a follow-up question after asking about the weather, not to mention even more complex questions, which they could not handle at all. ChatGPT and other Large Language Models (LLMs) turned that in the meantime upside down. They allow fluent and continuous conversations on a human-like level with very complex sentences and diffused in the meantime into all kinds of applications and areas. One area that was not spared from this development, is the medical domain. An indicator for this is the medical search engine PubMed, which comprises currently more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books. As of March 2024, the search term “ChatGPT” already returns over 2,700 results. In general, it takes some time, until reviews, and especially systematic reviews appear for a “new” topic or discovery. However, not for ChatGPT, and the additional search restriction to “systematic review” for article type under PubMed, returns still 31 contributions, as of March 19 2024. After filtering out non-systematic reviews from the returned results, 19 publications are included. In this meta-review, we want to take a closer look at these contributions on a higher level and explore the current evidence of ChatGPT in the medical domain, because systematic reviews belong to the highest form of knowledge in science.

Introduction: Addressing the Information Deluge

Since ChatGPT’s release in late 2022, the medical literature has been flooded with over 2,700 papers exploring its utility. For a researcher or clinician, keeping up with this firehose of information is nearly impossible. While individual systematic reviews have attempted to synthesize these findings, they often focus on narrow niches, such as neurosurgery or dental education.

This paper presents a "meta-review"—a systematic review of existing systematic reviews—to provide a high-level status quo of ChatGPT in the entire healthcare sector. By identifying common trends, performance benchmarks, and pressing challenges, the authors evaluate the feasibility of moving these models from the lab to the clinic.

Methodology: A Systematic Approach

The researchers employed a methodology based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. The search strategy involved querying PubMed using the term "ChatGPT" and filtering specifically for the "Systematic Review" article type.

PRISMA flow diagram illustrating the identification, screening, and inclusion of studies for the meta-review.

Out of 31 initial results, two authors independently screened the papers. They excluded non-systematic reviews, editorials, comments, and non-English publications. This "meta" approach treats existing peer-reviewed syntheses as primary data points, rather than individual clinical trials, to extract overarching outcomes and challenges.

Data Synthesis and Key Insights

The final 19 systematic reviews cover a wide range of medical domains, including neurosurgery, radiology, dentistry, pediatrics, and gastroenterology. Applications analyzed include medical education, scientific writing, patient communication, and clinical examination performance (e.g., USMLE-style tests).

Key insights from the synthesis include:

  • The Speed of Science: Traditionally, systematic reviews take years to emerge. For ChatGPT, high-quality reviews appeared within months, reflecting the unprecedented pace of AI research.
  • The "Assistant" vs. "Expert" Paradox: ChatGPT is widely praised as a helpful assistant for scientific writing and administrative tasks. However, it is explicitly noted as "not fully capable" of conducting its own critical systematic reviews because it fails to follow rigorous PRISMA protocols.
  • Performance vs. Difficulty: ChatGPT’s performance is inversely proportional to question difficulty. While it can pass general medical exams, its reliability drops significantly when faced with complex, ambiguous, or highly specialized clinical cases.

Key Results and Outcomes

The meta-review highlights several quantitative and qualitative benchmarks across the included studies.

Summary of selected systematic reviews, including publication dates and key outcomes regarding ChatGPT's performance in various medical fields.
  • Exam Performance: Multiple reviews confirm that ChatGPT can achieve a passing grade on various medical board examinations.
  • Radiology Workflows: Research indicates potential for improving the efficiency of radiological reporting and diagnosis accuracy.
  • The Hallucination Hurdle: Across almost all 19 reviews, "hallucination" (generating confident but false information) remains the primary barrier to clinical adoption.
  • Human-in-the-Loop: The consensus is that ChatGPT should currently be used with caution and always under the supervision of a human expert.

Limitations and Future Outlook

The authors identify three main roadblocks for the future of LLMs in medicine:

  1. Specialty-Specific Fine-Tuning: Future work must focus on training models on curated, high-quality medical literature rather than the open web to improve accuracy in niche domains.
  2. Lack of Unified Evaluation: There is currently no "gold standard" for testing an LLM’s clinical performance. The authors call for a unified evaluation system that goes beyond simple multiple-choice questions.
  3. Ethical and Legal Frameworks: Issues regarding patient privacy, data security, and the "authorship" of AI-generated medical content remain unresolved.

The paper concludes that a collaborative effort between NLP researchers, clinicians, and legislators is required to create safe, "safe-by-design" AI tools for patient care.

Summary updated

Abstract

Abstract Since its release at the end of 2022, ChatGPT has seen a tremendous rise in attention, not only from the general public, but also from medical researchers and healthcare professionals. ChatGPT definitely changed the way we can communicate now with computers. We still remember the limitations of (voice) assistants, like Alexa or Siri, that were “overwhelmed” by a follow-up question after asking about the weather, not to mention even more complex questions, which they could not handle at all. ChatGPT and other Large Language Models (LLMs) turned that in the meantime upside down. They allow fluent and continuous conversations on a human-like level with very complex sentences and diffused in the meantime into all kinds of applications and areas. One area that was not spared from this development, is the medical domain. An indicator for this is the medical search engine PubMed, which comprises currently more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books. As of March 2024, the search term “ChatGPT” already returns over 2,700 results. In general, it takes some time, until reviews, and especially systematic reviews appear for a “new” topic or discovery. However, not for ChatGPT, and the additional search restriction to “systematic review” for article type under PubMed, returns still 31 contributions, as of March 19 2024. After filtering out non-systematic reviews from the returned results, 19 publications are included. In this meta-review, we want to take a closer look at these contributions on a higher level and explore the current evidence of ChatGPT in the medical domain, because systematic reviews belong to the highest form of knowledge in science.
BibTeX
@misc{egger2024medical,
  author = {Jan Egger and Malik Sallam and Gijs Luijten and Christina Gsaxner and Antonio Pepe and Jens Kleesiek and Behrus Puladi and Jianning Li},
  title = {{Medical ChatGPT – A systematic Meta-Review}},
  year = {2024},
  doi = {10.1101/2024.04.02.24304716},
  howpublished = {medRxiv},
}
Figure