TL;DR
This paper tackles the time-consuming task of generating discharge summary sections for electronic health records by fine-tuning various large language models (LLMs). They found that smaller encoder-decoder models can perform comparably to larger decoder-only models and that providing the preceding content of the discharge note as context is sufficient for good performance.
What Problem Does This Paper Address?
Clinicians face a significant burden from clinical documentation, which takes time away from patient care and can lead to burnout. Discharge summaries, in particular, require synthesizing a lot of information into concise sections like the "brief hospital course" and "discharge instructions." This task is time-consuming and challenging to do manually. The BioNLP 2024 Shared Task "Discharge Me!" aims to automate the generation of these sections using LLMs to alleviate this burden.
Methods
The researchers experimented with fine-tuning several open-source LLMs, including both decoder-only models (Llama3, Mistral, Gemma) and encoder-decoder models (PRIMERA, LongT5). They explored different input contexts, including the base discharge note content and optionally including radiology reports. They also investigated various decoding strategies, model ensembling, and adapter merging techniques. A key aspect of their approach was fine-tuning these models on the MIMIC-IV dataset, specifically focusing on generating the "brief hospital course" and "discharge instruction" sections. For decoder-only models, they used LoRA (Low-Rank Adaptation) for efficient fine-tuning.
Datasets
The primary dataset used was the MIMIC-IV database, which provided 109,168 patient discharge summaries. This dataset was split into training, validation, and two phases of testing. The researchers focused on using the free-text clinical notes within these discharge summaries as input.
Key Insights and Ideas
- Context is King (and Sufficient): The preceding content of the discharge note itself (before the target section) proved to be sufficient context for generating high-quality "brief hospital course" and "discharge instruction" sections. Including additional information like radiology reports did not improve performance and sometimes even degraded it.
- Smaller Models Can Shine: Surprisingly, smaller encoder-decoder models like PRIMERA performed comparably to, and sometimes slightly better than, larger decoder-only models (like Llama3, Mistral, Gemma) when fine-tuned with LoRA. This suggests that architectural choices and fine-tuning strategies can be more impactful than sheer model size for this specific task.
- Specialization Matters: Training separate models for the "brief hospital course" and "discharge instruction" sections yielded better results than training a single unified model for both. This highlights the importance of task-specific fine-tuning.
- Instruction Tuning Can Harm: The instruction-tuned versions of decoder-only models performed worse than their base counterparts, aligning with existing research that suggests instruction tuning can sometimes negatively impact performance on specific NLP benchmarks.
Key Results
The team's best submission, using PRIMERA, ranked 3rd overall on the final leaderboard, achieving strong results in both automatic and manual evaluations. While specific quantitative scores are detailed in tables, a key takeaway is that their fine-tuned models significantly outperformed out-of-the-box instruction-tuned models in a zero-shot setting. They also found that advanced decoding methods like beam search and contrastive search did not consistently improve results over the baseline greedy search for their task.
Limitations and Future Work
The authors acknowledge that they did not explore all possible model ensembling or interpolation techniques. They also note that the generalizability of their models to generating other types of clinical notes or notes from different institutions remains to be studied. Future work could investigate how to better select input context, potentially exploring de-duplication techniques to reduce noise. Further research is also needed to understand how these generation systems can be integrated into clinical workflows and how they impact clinicians in practice. They also plan to investigate potential biases and overfitting in their models.
Summary updated