Clinical knowledge in LLMs does not translate to human interactions

Andrew M. Bean, Rebecca Payne, Guy Parsons, Hannah Rose Kirk, Juan Ciro, Rafael Mosquera, Sara Hincapié Monsalve, Aruna S. Ekanayaka, Lionel Tarassenko, Luc Rocher, Adam MahdiarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) have achieved remarkable scores on medical benchmarks, yet their utility in real-world patient interactions remains largely untested. This study investigates the "interaction gap" between a model's internal clinical knowledge and its ability to assist laypeople in identifying medical conditions and appropriate next steps. Through a large-scale randomized controlled trial (n=1,298), we find that while LLMs excel in direct-query scenarios, their performance drops significantly when mediated by human users. Participants using LLMs were less accurate at identifying conditions than those using standard internet searches, highlighting that current benchmarks are poor predictors of real-world medical safety and utility.

Introduction

The rapid advancement of "Doctor-in-a-box" AI applications, powered by models like GPT-4o and Med-PaLM 2, has sparked hope for democratizing medical expertise. These models frequently demonstrate "superhuman" performance on static benchmarks such as the US Medical Licensing Exam (USMLE). However, medical care is an interactive process. Most existing evaluations rely on "direct prompting"—providing the AI with a complete case file—or "AI-to-AI simulations," where one model acts as a patient. These methods fail to account for the messy, idiosyncratic way real humans describe symptoms and interpret advice. This research addresses a critical question: Does high clinical knowledge translate to better health outcomes for laypeople?

Methods

Researchers at the University of Oxford and several AI labs conducted a large-scale Randomized Controlled Trial (RCT) involving 1,298 participants from the UK, stratified to match national demographics.

Overview of the experimental design, including scenario creation, gold standard definition, participant assignment to LLM or control arms, and the scoring of responses.

The study design consisted of four primary components:

  1. Medical Scenarios: Three doctors drafted 10 detailed vignettes covering a range of conditions, from the common cold to life-threatening emergencies like subarachnoid hemorrhage.
  2. The Task: Participants were assigned a scenario and asked to identify the likely condition and the correct "disposition" (e.g., calling an ambulance, visiting a GP, or self-care).
  3. Experimental Arms: Participants were randomly assigned to use one of three LLMs (GPT-4o, Llama 3 70B, or Command R+) or a control group instructed to use their own knowledge or standard internet searches.
  4. Evaluation Baselines: To benchmark human-AI performance, the researchers also tested the LLMs "alone" (direct query with the full scenario) and used LLM-to-LLM "simulated patient" interactions.

Gold standard diagnoses and dispositions were established by a panel of seven doctors.

Results

The study revealed a significant "Human-in-the-Loop" penalty. While LLMs performed exceptionally well in isolation, their utility plummeted during human interaction.

The Information Bottleneck

The "signal" in medical communication is lost at two points. First, human users often omit critical diagnostic details (e.g., the sudden onset of a headache). Second, LLMs frequently fail to ask clarifying questions, instead providing generic advice or hallucinating based on incomplete input.

Comparative Performance

  • Condition Identification: Participants using LLMs correctly identified the medical condition in less than 34.5% of cases. Surprisingly, the control group (using standard search) performed better, with a 47.0% accuracy rate.
  • Disposition Accuracy: There was no statistically significant difference between the LLM groups and the control group in choosing the correct course of action (roughly 43-44% accuracy for all).
  • Model Superiority (Alone): When queried directly with the full case file, LLMs achieved 94.9% accuracy for conditions and 56.3% for dispositions.

The Failure of Simulations

The research found that LLM-simulated patients are "too good." They provide structured, relevant information that real humans do not, leading to inflated performance scores that fail to generalize to real-world interactions. Furthermore, high scores on the MedQA (USMLE) benchmark were almost entirely uncorrelated with a model's ability to help a human user in a live chat.

Discussion and Future Work

The findings suggest that clinical knowledge alone is insufficient for medical AI safety. For these tools to be effective, models must become proactive. Rather than waiting for a perfect prompt, the AI must lead a clinical interview, actively requesting missing information to narrow down a diagnosis—mimicking the behavior of a human physician.

The authors conclude that "systematic human user testing" must become a mandatory part of the AI safety pipeline. Relying on static benchmarks or AI simulations creates a false sense of security that does not hold up when real people are in the loop. Future research should explore advanced techniques like Chain-of-Thought or specialized medical fine-tuning to bridge this interaction gap.

Summary updated

Abstract

Global healthcare providers are exploring use of large language models (LLMs) to provide medical advice to the public. LLMs now achieve nearly perfect scores on medical licensing exams, but this does not necessarily translate to accurate performance in real-world settings. We tested if LLMs can assist members of the public in identifying underlying conditions and choosing a course of action (disposition) in ten medical scenarios in a controlled study with 1,298 participants. Participants were randomly assigned to receive assistance from an LLM (GPT-4o, Llama 3, Command R+) or a source of their choice (control). Tested alone, LLMs complete the scenarios accurately, correctly identifying conditions in 94.9% of cases and disposition in 56.3% on average. However, participants using the same LLMs identified relevant conditions in less than 34.5% of cases and disposition in less than 44.2%, both no better than the control group. We identify user interactions as a challenge to the deployment of LLMs for medical advice. Standard benchmarks for medical knowledge and simulated patient interactions do not predict the failures we find with human participants. Moving forward, we recommend systematic human user testing to evaluate interactive capabilities prior to public deployments in healthcare.
BibTeX
@misc{bean2025clinical,
  author = {Andrew M. Bean and Rebecca Payne and Guy Parsons and Hannah Rose Kirk and Juan Ciro and Rafael Mosquera and Sara Hincapié Monsalve and Aruna S. Ekanayaka and Lionel Tarassenko and Luc Rocher and Adam Mahdi},
  title = {{Clinical knowledge in LLMs does not translate to human interactions}},
  year = {2025},
  eprint = {2504.18919},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.18919},
}
Figure