A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text Summarisation

Elham Asgari, Nina Montaña-Brown, Magda Dubois, Saleh Khalil, Jasmine Balloch, Dominic Pimenta

TL;DR

Researchers have developed a "clinician-in-the-loop" framework to systematically catch and categorize LLM errors in medical summaries. By moving beyond standard text-similarity metrics and using a safety-first taxonomy, they demonstrated that iterative prompt engineering can reduce major LLM errors to rates lower than those found in human-written clinical notes.

What Problem Does This Paper Address?

Clinicians are drowning in paperwork, and while Large Language Models (LLMs) offer a lifeline for automating consultation summaries, they bring a dangerous side effect: hallucinations. In a medical context, a "hallucination" isn't just a quirk; it’s a potential patient safety incident.

The authors identify a major gap in how we evaluate these models. Standard NLP metrics like ROUGE or BLEU measure how much the output looks like a reference text, but they are blind to clinical risk. For example, missing the word "not" in "patient does not have a penicillin allergy" is a tiny change in a ROUGE score but a catastrophic failure in clinical safety. This paper introduces a way to quantify these risks through a clinically-informed lens.

Methods

The authors introduced a multi-component framework centered on CREOLA, a custom graphical user interface designed for clinicians to review and label LLM outputs. The key technical contributions include:

  • Clinical Error Taxonomy: Instead of generic "hallucinations," they categorized errors into Fabrication (making things up), Negation (flipping a fact), Causality (assuming a cause), and Contextual (mixing up topics). Omissions were similarly categorized by clinical section (e.g., missing past medical history vs. missing the treatment plan).
  • Safety Risk Matrix: Borrowing from medical device certification standards, they mapped errors onto a 5x5 grid of Likelihood vs. Consequence, ranging from "Minor" to "Catastrophic."
  • Iterative Experimental Design: They ran 18 distinct experimental configurations, testing variables like prompt structure, "atomization" (breaking transcripts into facts first), and function calling (forcing JSON output).
  • Clinician-in-the-loop: 50 medical doctors were recruited to manually evaluate nearly 13,000 sentences, ensuring the "ground truth" was determined by medical experts, not just automated scripts.

Datasets

  • PriMock: The primary dataset used, consisting of primary care consultation transcripts.
  • ACI Bench: A publicly available clinical dataset used for final validation (Experiment 18).
  • Scale: The study involved 450 transcript-note pairs, totaling 49,590 transcript sentences and 12,999 clinical note sentences.

Key Insights and Ideas

  • The "Atomization" Paradox: One might assume that asking an LLM to extract a list of facts before writing a summary (a form of Chain-of-Thought) would improve accuracy. Surprisingly, the authors found this "atomization" step actually increased major hallucinations and omissions.
  • Negation is the Most Dangerous Hallucination: 30% of hallucinations were negations. These were particularly prevalent in the "Plan" section of notes, where they could lead to direct harm by contradicting a clinician's actual instructions.
  • Structure Beats Free-Text: Moving from free-text prompts to structured JSON outputs via "function calling" significantly improved reliability. It forced the model to adhere to specific subheadings and reduced the likelihood of the model "wandering" off-script.
  • The Human Baseline: In a head-to-head comparison (Experiment 17), human-written notes actually contained more hallucinations than the optimized LLM, though humans were better at avoiding omissions. This suggests that a well-tuned LLM might actually be a "safer" documenter than a tired clinician.

Key Results

  • Baseline Rates: Initially, the LLM showed a 1.47% hallucination rate (44% of which were "major") and a 3.45% omission rate (17% "major").
  • Optimization Success: Through iterative refinement (specifically Experiment 11 using function calling), the team eliminated major omissions entirely and reduced the total number of hallucinations by 25%.
  • Sub-Human Error Rates: The best-performing LLM configurations achieved error rates lower than those previously reported for human-written notes (which typically average 1 error and 4 omissions per note).

Limitations and Future Work

  • Sample Size: While the sentence count was high, the number of unique transcripts (25 per experiment) was relatively small, which may limit the diversity of clinical scenarios captured.
  • Single Model Focus: The study relied exclusively on OpenAI’s GPT-4. The authors acknowledge that open-source models (like Llama-3 or Mistral) might behave differently under this framework.
  • Scalability: Manual clinician review is expensive and slow. The authors suggest that "LLM-as-a-Judge"—using a second LLM to perform the safety audit—is a necessary next step for scaling this framework to larger datasets.
  • Prompt Sensitivity: The results showed that even minor changes in prompt "style" could significantly swing error rates, highlighting the fragility of current LLM deployments in high-stakes environments.

Summary updated

Abstract

Abstract Integrating large language models (LLMs) into healthcare settings can improve workflow efficiency and patient care by automating tasks such as summarising consultations. However, ensuring the fidelity between LLM outputs and ground truth information is crucial, as errors can lead to miscommunication between patients and clinicians, resulting in incorrect diagnoses, treatment decisions and compromised patient safety. We introduce a clinician-in-the-loop framework with: 1) a clinically and technically-informed error taxonomy to classify LLM outputs, 2) an experimental structure to comprehensively and iteratively compare outputs within our LLM document generation pipeline, 3) a clinical safety framework to assess potential harms of errors in LLM outputs, and 4) an encompassing graphical user interface (GUI), CREOLA, to perform and assess all previous steps. Our clinical error metrics were derived from 18 experimental configurations involving LLMs for clinical note generation consisting of 49,590 transcript and 12,999 clinical note sentences. Overall, we observed a 1.47% hallucination rate (44% rated ‘major’) and a 3.45% omission rate (17% ‘major’). Through iterative prompts and workflow refinements, we reduced major errors below previously reported human note-taking error rates, underscoring the potential of our framework to enable safer clinical documentation.
BibTeX
@article{asgari2025framework,
  author = {Elham Asgari and Nina Montaña-Brown and Magda Dubois and Saleh Khalil and Jasmine Balloch and Dominic Pimenta and Dominic Pimenta},
  title = {{A Framework to Assess Clinical Safety and Hallucination Rates of LLMs for Medical Text Summarisation}},
  journal = {Npj Digit. Medicine},
  year = {2025},
  doi = {10.1038/s41746-025-01670-7},
}
Figure