TL;DR
This paper presents a Retrieval-Augmented Generation (RAG) system for automating the generation of critical sections in hospital discharge summaries. By combining a Large Language Model (LLM) with retrieved examples of similar discharge notes, their system aims to improve accuracy and reduce hallucinations, outperforming zero-shot baselines on the "Discharge Me!" task.
What Problem Does This Paper Address?
Generating detailed clinical notes in Electronic Health Records (EHRs) is a significant burden on healthcare professionals, contributing to burnout and operational inefficiencies. The "Discharge Me!" task aims to automate the creation of crucial discharge summary sections, specifically the "Brief Hospital Course" and "Discharge Instructions." While LLMs have advanced NLP capabilities, they are prone to generating inaccurate or fabricated information (hallucinations), especially when encountering data outside their training distribution. This paper tackles the challenge of producing reliable and accurate discharge summaries by leveraging RAG to ground LLM generation with external knowledge.
Methods
The core of the proposed system is a Retrieval-Augmented Generation (RAG) framework. The process begins with generating structured JSON summaries from the original patient discharge notes using an LLM (Mistral-7B-Instruct-v0.2). This structured data, which includes patient demographics, medical history, and other key information, is then used to guide the generation of the target sections.
The RAG approach involves two main steps:
- Retrieval: For a given patient's structured summary, the system retrieves relevant examples of "Brief Hospital Course" and "Discharge Instructions" from the training dataset. Two retrieval methods were explored:
- BM25: A traditional keyword-based retrieval method.
- SentenceBERT (SBERT): A neural network-based method that captures semantic similarity by encoding text into dense vector representations. The paper specifically uses a pre-trained SBERT model fine-tuned on medical text.
- Generation: The retrieved examples, along with the patient's structured summary, are provided as context to a frozen LLM. This "k-shot" prompting aims to guide the LLM in generating accurate and contextually relevant discharge summary sections.
The authors also experimented with a zero-shot baseline, where the LLM generated the sections directly from the structured JSON without any retrieval.
Datasets
The study utilizes the dataset provided for the BioNLP ACL'24 "Discharge Me!" shared task. This dataset is derived from the MIMIC-IV-Note and MIMIC-IV-ED submodules and contains approximately 109,168 Emergency Department (ED) visits. Each visit includes chief complaints, ICD diagnosis codes, radiology reports, and the full discharge summary text. The dataset is split into training, validation, and two testing phases.
Key Insights and Ideas
- RAG is crucial for improving LLM performance in clinical text generation: The significant performance uplift observed when moving from a zero-shot approach to RAG highlights the importance of providing external, relevant context to LLMs for specialized tasks like clinical documentation.
- Semantic similarity retrieval (SBERT) outperforms keyword-based retrieval (BM25): The results suggest that understanding the meaning of the text, rather than just keyword matching, is more effective for retrieving relevant examples that can guide LLM generation in this domain.
- Structured patient summaries reduce noise: Pre-processing the raw discharge notes into structured JSON format appears to help the LLM focus on essential information and potentially mitigate hallucinations.
- Full discharge summary as input can be beneficial: Preliminary results with using the full discharge summary (FDS) as input to the RAG system showed promising improvements, suggesting that more comprehensive input can lead to better generation.
Key Results
The paper evaluated four systems: PS+Zero-shot, PS+SBERT-RAG, PS+BM25-RAG, and FDS+SBERT-RAG.
- The top-performing system, PS+SBERT-RAG, achieved an overall score of 0.183 at the competition deadline.
- The PS+Zero-shot baseline achieved an overall score of 0.172, demonstrating a performance gain of approximately 6.4% with the RAG approach.
- PS+SBERT-RAG also performed favorably in semantic similarity metrics (BERTScore: 0.282, Meteor: 0.284) and factual alignment (AlignScore: 0.21, MEDCON: 0.215).
- Preliminary results for FDS+SBERT-RAG showed even better performance across several metrics, suggesting that providing more input context is beneficial.
Limitations and Future Work
The authors acknowledge several limitations:
- Maximum sequence length constraints: The retrieval encoder (SentenceBERT) had a 350-token limit, leading to the loss of contextual information from longer discharge summaries. This also impacted the LLM's context window.
- Underutilization of complementary data: The system did not effectively leverage available radiology reports and ICD-9/10 diagnosis codes, which could enhance understanding and generation.
- Ad-hoc prompt design: The prompts were created without direct medical professional guidance, potentially lacking necessary context and domain-specific nuances.
- Lack of domain adaptation: The LLM and SBERT retrieval model were not specifically fine-tuned on the clinical domain, which could lead to issues with terminology and concept understanding.
Future work directions include:
- Fine-tuning: Fine-tuning the pipeline on "Brief Hospital Course" and "Discharge Instructions" sections.
- Constrained decoding/post-processing: To improve n-gram overlap with reference summaries.
- Stylistic similarity retrieval: To indirectly benefit n-gram metrics.
- Integrating UMLS: Retrieving Unified Medical Language System (UMLS) concept-rich examples or integrating UMLS databases to improve metrics like MEDCON.
- Advanced RAG architectures: Exploring iterative retrieval and multi-step reasoning to overcome Naive RAG limitations.
- Leveraging more data: Effectively integrating radiology reports and diagnosis codes.
- Domain-specific prompt engineering: Collaborating with medical professionals to design better prompts.
Summary updated