Guideline-enhanced large language models outperform physician-test takers on EASL Campus quizzes multiple choice questions

Mauro Giuffrè, Alfred Distefano, Simone Kresevic, Lory Saveria Crocè, Dennis Shung, Miloš Ajčević, M Gulotta, F. Orbosuè, Lisa Fusaro, Simone Giovanni ErmacoraJHEP 2025 · DOI · OpenAlex

Abstract

The recent progress in Large Language Models (LLMs) has garnered widespread attention in the medical community, especially given their ability to handle multimodal data such as images or audio-recordings. Safe deployment into clinical practice remains unclear, with a recent systematic review identifying a broad range of accuracy (6.4-91.4%) when ChatGPT answered clinical questions on digestive disease-related topics1. This wide range of performance can be attributed to the use of baseline models without incorporating external medical knowledge from relevant medical guidelines2-4.
BibTeX
@inproceedings{giuffre2025guideline,
  author = {Mauro Giuffrè and Alfred Distefano and Simone Kresevic and Lory Saveria Crocè and Dennis Shung and Miloš Ajčević and M Gulotta and F. Orbosuè and Lisa Fusaro and Simone Giovanni Ermacora and Dennis L. Shung},
  title = {{Guideline-enhanced large language models outperform physician-test takers on EASL Campus quizzes multiple choice questions}},
  booktitle = {JHEP},
  year = {2025},
  doi = {10.1016/j.jhepr.2025.101523},
}

Loading related content...

Figure