A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation

Elham Asgari, Nina Montaña Brown, Magda Dubois, Saleh Khalil, Jasmine Balloch, Joshua Au Yeung, Dominic Pimenta

TL;DR

This paper introduces a comprehensive framework to quantify and mitigate the clinical risks of using Large Language Models (LLMs) for medical note summarization. By combining a specialized clinical error taxonomy with an iterative prompt-engineering workflow and a custom annotation tool (CREOLA), the researchers demonstrated that LLMs can be tuned to produce documentation with error rates lower than those of human clinicians.

What Problem Does This Paper Address?

Clinician burnout is a systemic crisis, driven largely by the heavy administrative burden of electronic health record (EHR) documentation. While LLMs offer a promising solution for automating consultation summaries, they suffer from two critical flaws: hallucinations (generating false information) and omissions (missing vital clinical details).

Standard NLP metrics like ROUGE or BLEU are insufficient here because they measure textual similarity rather than clinical safety. A "minor" textual change in a medical note—like swapping "no fever" for "fever"—can have catastrophic consequences for patient care. This paper addresses the lack of a standardized, safety-first framework for evaluating and refining LLM outputs in a high-stakes clinical environment.

Methods

The authors propose a four-pillar framework:

  1. Clinical Error Taxonomy: A granular system for classifying errors into hallucinations (Fabrication, Negation, Causality, Contextual) and omissions (Current issues, History, Plan).
  2. Iterative Experimental Structure: A process of running multiple "experiments" (18 in total) to test how different prompt strategies and workflows affect error rates.
  3. Clinical Safety Framework: A risk-grading system inspired by medical device certification that classifies errors as "Major" (potentially impacting diagnosis or management) or "Minor."
  4. CREOLA GUI: A bespoke graphical user interface designed for clinicians to efficiently label and consolidate reviews of LLM-generated text.

The team tested various technical approaches, including atomization (breaking transcripts into atomic facts via Chain-of-Thought), structured prompting (using SOAP note formats), and function calling to ensure outputs adhered to strict JSON schemas for EHR integration.

Datasets

  • PriMock: 25 primary care consultation transcripts used as the primary dataset for the 18 iterative experiments.
  • ACI Bench: A publicly available dataset used for final performance validation (Experiment 18).
  • Expert Annotation: The study involved a massive manual evaluation effort, with 50 medical doctors annotating 12,999 sentences to provide a "ground truth" for clinical safety.

Key Insights and Ideas

  • Omissions are Common, Hallucinations are Dangerous: While omissions occurred more frequently (3.45% vs. 1.47% for hallucinations), hallucinations were far more likely to be classified as "Major" risks (44% of hallucinations vs. 16.7% of omissions).
  • The "Plan" Section is High-Risk: Hallucinations were most frequent in the "Plan" section of the notes (21%). This is particularly concerning because this section contains direct instructions for patient management.
  • The Danger of Negations: Negation errors—where the model incorrectly states a symptom is absent—represented 30% of hallucinations. These are difficult for readers to spot without the full transcript and can lead to significant diagnostic delays.
  • Workflow over Raw Power: The study found that simply using a powerful model isn't enough. Iterative refinements, such as instructing the model to output "unknown" for missing information rather than guessing, were essential to reaching sub-human error rates.

Key Results

  • Baseline Performance: The researchers observed an initial 1.47% hallucination rate and a 3.45% omission rate across their experimental configurations.
  • Safety Gains: Through iterative prompt engineering (moving from Experiment 3 to Experiment 8), they reduced major hallucinations by 75% (from 4 to 1) and major omissions by 58% (from 24 to 10).
  • Surpassing Humans: In their best-performing iterations, the LLM produced fewer errors per note than the average reported for human-generated clinical notes (which typically contain at least 1 error and 4 omissions).

Limitations and Future Work

  • Sample Size: The study used a relatively small set of 25 transcripts per experiment to manage the high cost and time of expert clinician annotation.
  • Model Specificity: The evaluation was limited to OpenAI’s GPT-4. Future work should explore how open-source models or smaller, specialized medical LLMs perform under this framework.
  • Scalability: Manual human annotation is the "gold standard" but is not sustainable for large-scale deployment. The authors suggest that "LLM-as-a-Judge" (using a second LLM to audit the first) is a necessary next step to make this safety framework scalable.

Summary updated

Abstract

Abstract Integrating large language models (LLMs) into healthcare settings can improve workflow efficiency and patient care by automating tasks such as summarising consultations. However, ensuring the fidelity between LLM outputs and ground truth information is crucial, as errors can lead to miscommunication between patients and clinicians, resulting in incorrect diagnoses, treatment decisions and compromised patient safety. We introduce a clinician-in-the-loop framework with: 1) a clinically and technically-informed error taxonomy to classify LLM outputs, 2) an experimental structure to comprehensively and iteratively compare outputs within our LLM document generation pipeline, 3) a clinical safety framework to assess potential harms of errors in LLM outputs, and 4) an encompassing graphical user interface (GUI), CREOLA, to perform and assess all previous steps. Our clinical error metrics were derived from 18 experimental configurations involving LLMs for clinical note generation consisting of 49,590 transcript and 12,999 clinical note sentences. Overall, we observed a 1.47% hallucination rate (44% rated ‘major’) and a 3.45% omission rate (17% ‘major’). Through iterative prompts and workflow refinements, we reduced major errors below previously reported human note-taking error rates, underscoring the potential of our framework to enable safer clinical documentation.
BibTeX
@article{asgari2025framework,
  author = {Elham Asgari and Nina Montaña Brown and Magda Dubois and Saleh Khalil and Jasmine Balloch and Joshua Au Yeung and Dominic Pimenta},
  title = {{A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation}},
  journal = {Npj Digit. Medicine},
  year = {2025},
  doi = {10.1038/s41746-025-01670-7},
}
Figure