TL;DR
This paper tackles the challenge of automatically generating discharge summaries, a crucial but time-consuming task for physicians. They explore various NLP models, including fine-tuned BioBART and large language models like GPT-3.5, and investigate methods for incorporating structured EHR data and external knowledge to improve summary quality. Their best-performing model, a fine-tuned BioBART, achieved second place in the BioNLP 2024 "Discharge Me!" challenge.
What Problem Does This Paper Address?
Discharge summaries are vital documents that bridge the gap between inpatient care and a patient's subsequent treatment (rehabilitation, outpatient, etc.). They contain essential information like next steps for the patient, follow-up on incidental findings, new treatment plans, and more. However, creating these summaries is a significant burden on physicians, contributing to burnout. The authors aim to alleviate this by developing automated methods for generating two key sections: the Brief Hospital Course (BHC) and Discharge Instructions. The BHC summarizes the patient's journey from ED admission to discharge, while Discharge Instructions provide patient-friendly guidance for post-discharge care.
Methods
The researchers explored several approaches:
- Encoder-Decoder Models: They fine-tuned pre-trained models like BioBART and Clinical-T5 on the task of generating BHCs from unstructured clinical notes. BioBART, a model continuously pre-trained on biomedical abstracts, showed strong performance.
- Large Language Models (LLMs): They evaluated a black-box GPT-3.5 Turbo model in a zero-shot setting for both BHC and discharge instruction generation.
- Structured Data Integration: To further reduce physician burden, they developed a pipeline to generate BHCs solely from structured Electronic Health Record (EHR) data. This involved converting structured data into "pseudo-SOAP notes" which were then fed to GPT-3.5.
- Constrained Generation: To improve the inclusion of critical clinical concepts, they experimented with a constrained beam search approach. This method injects external knowledge (identified using MedCat and UMLS concepts) into the generation process.
Datasets
The primary dataset used was derived from the MIMIC-IV database, specifically focusing on discharge summaries from 109,168 emergency department visits. This dataset includes both unstructured text (discharge summaries, chief complaints, radiology reports) and structured data (demographics, ED diagnoses, procedures, medications, lab values). A subset of 250 discharge summaries was used for standardized final evaluation.
Key Insights and Ideas
- BioBART's Strength: Fine-tuning a BioBART model, especially with a larger, shuffled training dataset, proved to be the most effective approach among the encoder-decoder models, leading to their second-place finish in the challenge.
- Structured Data Alone is Challenging: Generating BHCs solely from structured EHR data using GPT-3.5 did not outperform the best model trained on unstructured text. This suggests that while structured data is valuable, the nuances and narrative flow of unstructured notes are still critical for high-quality summaries.
- Constrained Generation Backfires: Surprisingly, the constrained beam search approach, intended to improve recall of clinical concepts, actually worsened performance. The authors hypothesize this could be due to the complexity of tuning additional hyperparameters.
- LLMs vs. Fine-tuned Models: While LLMs like GPT-3.5 showed promise, fine-tuned models (specifically BioBART) generally outperformed them on this task, especially when considering the overall evaluation metrics.
Key Results
- BHC Generation: The BioBART-Shuffled model, trained on an expanded dataset, achieved the best performance among encoder-decoder models and secured 2nd place in the challenge.
- Discharge Instructions: The BioBART model also outperformed GPT-3.5 for discharge instruction generation and was used for their challenge submission.
- Structured Data Pipeline: The GPT-3.5 pipeline using pseudo-SOAP notes from structured data did not offer improvement over the best fine-tuned model.
- Constrained Generation: This method performed worse than the vanilla BioBART model.
- Qualitative Evaluation: A clinician review of a sample of generations indicated that while completeness was generally good, readability and correctness varied. Discharge instructions received slightly higher ratings than BHCs, likely due to the inherent complexity of BHCs.
Limitations and Future Work
- Computational Constraints: The authors were unable to perform extensive hyperparameter tuning, which might have impacted the performance of certain methods, particularly constrained generation.
- Clinical Validity Nuances: The paper acknowledges that the current task framing doesn't fully automate the summarization process, as physicians still need to write the core notes. The format and stylistic choices in real-world clinical documentation also introduce significant variability.
- Future Directions: The authors suggest that further research into automated metrics for semantic comparison is needed. They also note the potential for improvement with more computational resources. The surprising underperformance of constrained generation warrants further investigation.
Summary updated