Abstract
Large Language Models (LLMs) have achieved remarkable scores on medical benchmarks, yet their utility in real-world patient interactions remains largely untested. This study investigates the "interaction gap" between a model's internal clinical knowledge and its ability to assist laypeople in identifying medical conditions and appropriate next steps. Through a large-scale randomized controlled trial (n=1,298), we find that while LLMs excel in direct-query scenarios, their performance drops significantly when mediated by human users. Participants using LLMs were less accurate at identifying conditions than those using standard internet searches, highlighting that current benchmarks are poor predictors of real-world medical safety and utility.
Introduction
The rapid advancement of "Doctor-in-a-box" AI applications, powered by models like GPT-4o and Med-PaLM 2, has sparked hope for democratizing medical expertise. These models frequently demonstrate "superhuman" performance on static benchmarks such as the US Medical Licensing Exam (USMLE). However, medical care is an interactive process. Most existing evaluations rely on "direct prompting"—providing the AI with a complete case file—or "AI-to-AI simulations," where one model acts as a patient. These methods fail to account for the messy, idiosyncratic way real humans describe symptoms and interpret advice. This research addresses a critical question: Does high clinical knowledge translate to better health outcomes for laypeople?
Methods
Researchers at the University of Oxford and several AI labs conducted a large-scale Randomized Controlled Trial (RCT) involving 1,298 participants from the UK, stratified to match national demographics.
The study design consisted of four primary components:
- Medical Scenarios: Three doctors drafted 10 detailed vignettes covering a range of conditions, from the common cold to life-threatening emergencies like subarachnoid hemorrhage.
- The Task: Participants were assigned a scenario and asked to identify the likely condition and the correct "disposition" (e.g., calling an ambulance, visiting a GP, or self-care).
- Experimental Arms: Participants were randomly assigned to use one of three LLMs (GPT-4o, Llama 3 70B, or Command R+) or a control group instructed to use their own knowledge or standard internet searches.
- Evaluation Baselines: To benchmark human-AI performance, the researchers also tested the LLMs "alone" (direct query with the full scenario) and used LLM-to-LLM "simulated patient" interactions.
Gold standard diagnoses and dispositions were established by a panel of seven doctors.
Results
The study revealed a significant "Human-in-the-Loop" penalty. While LLMs performed exceptionally well in isolation, their utility plummeted during human interaction.
The Information Bottleneck
The "signal" in medical communication is lost at two points. First, human users often omit critical diagnostic details (e.g., the sudden onset of a headache). Second, LLMs frequently fail to ask clarifying questions, instead providing generic advice or hallucinating based on incomplete input.
Comparative Performance
- Condition Identification: Participants using LLMs correctly identified the medical condition in less than 34.5% of cases. Surprisingly, the control group (using standard search) performed better, with a 47.0% accuracy rate.
- Disposition Accuracy: There was no statistically significant difference between the LLM groups and the control group in choosing the correct course of action (roughly 43-44% accuracy for all).
- Model Superiority (Alone): When queried directly with the full case file, LLMs achieved 94.9% accuracy for conditions and 56.3% for dispositions.
The Failure of Simulations
The research found that LLM-simulated patients are "too good." They provide structured, relevant information that real humans do not, leading to inflated performance scores that fail to generalize to real-world interactions. Furthermore, high scores on the MedQA (USMLE) benchmark were almost entirely uncorrelated with a model's ability to help a human user in a live chat.
Discussion and Future Work
The findings suggest that clinical knowledge alone is insufficient for medical AI safety. For these tools to be effective, models must become proactive. Rather than waiting for a perfect prompt, the AI must lead a clinical interview, actively requesting missing information to narrow down a diagnosis—mimicking the behavior of a human physician.
The authors conclude that "systematic human user testing" must become a mandatory part of the AI safety pipeline. Relying on static benchmarks or AI simulations creates a false sense of security that does not hold up when real people are in the loop. Future research should explore advanced techniques like Chain-of-Thought or specialized medical fine-tuning to bridge this interaction gap.
Summary updated