Detecting Omissions in LLM-Generated Medical Summaries

Achir Oukelmoun, Nasredine Semmar, Gaël de Chalendar, Clement Cormi, Mariame Oukelmoun, Eric Vibert, Marc-Antoine AllardEMNLP 2025

TL;DR

Large Language Models (LLMs) are increasingly used to summarize medical texts, but they can omit critical information, posing a risk to patient care. This paper introduces EmbedKDECheck, a novel, computationally frugal method for detecting these omissions in LLM-generated medical summaries. It also presents a new, expert-validated French dataset specifically designed for this task.

What Problem Does This Paper Address?

The rapid adoption of LLMs in healthcare for tasks like generating consultation summaries and medical reports brings significant benefits. However, a critical concern is the potential for these models to omit crucial information from the original source text. This is distinct from "hallucinations" (generating incorrect information) and can lead to incomplete medical records, potentially jeopardizing patient care and decision-making. In the French healthcare sector, strict privacy regulations limit the use of cloud-based LLM solutions, necessitating on-premise or hybrid architectures. This constraint, coupled with limited access to the most advanced LLMs, exacerbates the need for efficient, resource-light quality control mechanisms for LLM-generated medical texts. Existing evaluation frameworks often focus on hallucinations, leaving a gap in addressing omissions, especially in specialized domains like medicine.

Methods

The paper proposes EmbedKDECheck, a black-box, reference-free method for detecting omissions. It operates by:

  1. Embedding Extraction: Using lightweight, pre-trained word embeddings (FastText and Word2Vec, combined in FTW2V) to represent both the input medical report and its generated summary. These embeddings are then projected into a lower-dimensional space using Principal Component Analysis (PCA) for efficiency and stability.
  2. Kernel Density Estimation (KDE): Fitting a KDE model to the embeddings of the summary. This models the "normal density" of information present in the summary.
  3. Omission Scoring: Evaluating the embeddings of the original report within the KDE model derived from the summary. Tokens from the original report that fall into regions of unusually low density in the summary's embedding space are flagged as potential omissions. A global omission score is computed as the maximum token-level omission score.

This approach is "frugal" because it avoids direct LLM interaction for evaluation, relies on lightweight embeddings, and can run on standard CPUs, making it suitable for resource-constrained environments.

Datasets

The authors introduce a new, synthetic dataset specifically for evaluating omission detection in French medical summaries.

  • Creation: The dataset was built from 50 anonymized medical reports. GPT-4-32K was used to generate 15 synthetic variants per report, aiming to mimic authentic clinical narratives while preserving privacy. To ensure significant omissions for evaluation, approximately 50% of sentences were removed from the original reports before generating summaries.
  • Content: The dataset contains 1,348 triplets, each comprising a medical report, a generated summary, and a binary label indicating whether the summary contains omissions.
  • Validation: The synthetic dataset was rigorously validated by two expert surgeons for similarity to real reports, linguistic diversity, and accuracy of the omission labels. Annotators achieved 95% accuracy and strong agreement (Cohen's κ = 0.82) in identifying omissions.

Key Insights and Ideas

  • Omissions are a distinct and critical risk: The paper clearly differentiates omissions from hallucinations, highlighting the unique danger they pose in medical contexts where missing information can directly impact patient care.
  • Frugality is key for medical LLM deployment: The authors emphasize the need for evaluation methods that are computationally inexpensive and can run locally, aligning with the practical constraints of healthcare IT infrastructure and privacy regulations.
  • KDE as an anomaly detector for semantic gaps: Using Kernel Density Estimation on embeddings is an innovative way to identify where information from the source text is "missing" from the summary's semantic space.
  • Expert validation is crucial for medical benchmarks: The paper underscores the importance of having medical professionals validate datasets to ensure they accurately reflect real-world scenarios and the criticality of omissions.

Key Results

  • EmbedKDECheck Performance: The proposed EmbedKDECheck achieved a high F1-Score of 0.91, with a recall of 0.88 and precision of 0.93 on the new French medical summary dataset.
  • Benchmarking: It outperformed other black-box methods like SelfCheckGPT and GPTScore, and struck a better balance than ChainPoll and G-Eval, which showed high recall but lower precision.
  • Computational Efficiency: EmbedKDECheck required an estimated 0.11 Tera FLOPS, significantly less than LLM-based evaluators (e.g., SelfCheckGPT-gpt4 at 8804.03 Tera FLOPS), demonstrating its frugality.
  • Dataset Reliability: Expert validation confirmed the reliability of the synthetic labels and the effectiveness of the omission scores in identifying responsible source tokens.

Limitations and Future Work

  • Embedding Quality: The effectiveness of EmbedKDECheck is dependent on the quality and domain relevance of the embeddings used. Fine-tuning or hybrid embedding strategies might be needed for highly specialized medical nuances.
  • Ground Truth Cost: Creating a fully reliable ground truth dataset with manually verified omission counts is prohibitively expensive and time-consuming, necessitating the use of synthetic data as a compromise.
  • Subtle Omissions: The method might face challenges with omissions that are subtle, stylistic, or semantically diffuse, especially if the omission rate used for dataset generation differs significantly.
  • Future Work: The authors suggest exploring interpretable, hybrid systems, extending the method to other forms of hallucinations, and adapting it to multilingual and cross-domain contexts (e.g., legal, finance).

Summary updated

BibTeX
@inproceedings{oukelmoun2025detecting,
  author = {Achir Oukelmoun and Nasredine Semmar and Gaël de Chalendar and Clement Cormi and Mariame Oukelmoun and Eric Vibert and Marc-Antoine Allard},
  title = {{Detecting Omissions in LLM-Generated Medical Summaries}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure