UoG Siephers at “Discharge Me!”: Exploring Ways to Generate Synthetic Patient Notes From Multi-Part Electronic Health Records

Erlend Frayling, Jake Lever, Graham McDonaldACL 2024

TL;DR

This paper explores how to best select and encode sections of Electronic Health Records (EHRs) to automatically generate discharge instructions and brief hospital course summaries. They found that using a single, well-chosen EHR section (specifically, the "History of Present Illness") often yields better results than combining multiple sections, challenging the intuition that more data is always better.

What Problem Does This Paper Address?

Writing comprehensive and accurate patient notes in Electronic Health Records (EHRs) is a time-consuming but crucial task for healthcare professionals. This burden can detract from direct patient care. The "Discharge Me!" shared task aims to automate the generation of two key components of EHRs: discharge instructions (guidelines for patients leaving the hospital) and brief hospital courses (summaries of a patient's stay). The challenge lies in effectively leveraging the vast and often disparate information within EHRs to produce these summaries. This paper specifically investigates how to best prepare and input EHR data into sequence-to-sequence (seq2seq) models for this generation task.

Methods

The researchers experimented with three main approaches for preparing EHR data for seq2seq models:

  • Separate Text Sections: This method involves using individual EHR sections as input. They focused on the "History of Present Illness" (HPI) and "Radiology Report" (RR) sections, hypothesizing that distinct sections might offer focused and coherent training data.
  • Concatenating Text Sections: Here, they combined the text of multiple EHR sections (HPI and RR) into a single input. To handle the longer input, they employed a model with a larger context window, specifically Longformer.
  • Concatenating Embedded Sections (Fusion Approach): This novel approach involves encoding each EHR section separately and then concatenating their encoded representations before passing them to the decoder. This is inspired by how readers process multiple pieces of information.

They trained two versions of each model: one for generating discharge instructions and another for brief hospital courses. Various pre-trained encoder-decoder models (T5, BART, RoBERTa, BERT, ClinicalBERT) and decoder-only models (GPT-2, Llama 3) were utilized.

Datasets

The primary dataset used was curated from the MIMIC-IV database, provided by the organizers of the "Discharge Me!" shared task. This dataset contains de-identified patient EHRs, including discharge summaries with various reports. For their experiments, they used a subset of 5000 random samples from the training set and 1000 from the validation set. The dataset includes gold-standard discharge instructions and brief hospital courses for evaluation.

Key Insights and Ideas

  • Less is More (Sometimes): The most surprising finding is that using a single, carefully selected EHR section (like HPI) often outperformed models that combined multiple sections. This suggests that simply feeding more data isn't always beneficial; the quality and relevance of the input section are paramount.
  • HPI is a Strong Contender: The "History of Present Illness" (HPI) section consistently proved to be a highly effective input for generating both discharge instructions and hospital course summaries, regardless of the model architecture. This section seems to capture a good balance of patient context and clinical detail.
  • Concatenation Challenges: Concatenating text sections (even with Longformer) didn't consistently yield better results than single sections. The "fusion" approach (concatenating embeddings) also didn't outperform the best single-section models, indicating that models might struggle to effectively integrate information from separately encoded sections.
  • Model Choice Matters: Different models showed varying preferences for input sections. While T5 and RoBERTa-GPT2 performed better with HPI, BERT-GPT2 and ClinicalBERT-GPT2 showed a slight preference for RR. However, the performance gains from HPI were generally more substantial.

Key Results

  • The T5 model trained on the HPI section achieved the best overall score (0.191) among encoder-decoder models.
  • The Llama 3 decoder-only model also demonstrated competitive performance, particularly when using the HPI section, rivaling the T5 model.
  • Performance gains from using HPI over RR were significant. For instance, the T5(HPI) model showed a 141% improvement in Overall Score compared to T5(RR).
  • Qualitative analysis of the best-performing model (T5(HPI)) showed that it could accurately describe core ailments and generate structured discharge instructions. However, follow-up treatments were sometimes inaccurate, and brief hospital courses sometimes included extended, but not strictly target, information.
  • A notable issue encountered was the model's attempt to generate English text even when the target section was in Spanish, highlighting a limitation in handling multilingual inputs.

Limitations and Future Work

  • Decoder Choice: The study primarily used GPT-2 as a decoder. The authors acknowledge that evaluating Llama 3 as a decoder in an encoder-decoder architecture could yield further insights, especially given Llama 3's strong performance as a decoder-only model.
  • Number of EHR Sections: The research focused on using one or two EHR sections. Future work could explore the impact of concatenating more EHR sections and investigate which other sections might be most beneficial as input.
  • Multilingual Support: The current models struggle with non-English text, indicating a need for multilingual capabilities in future research.
  • Fusion Approach Refinement: While the fusion approach showed promise, it did not outperform simpler methods. Further research into how to effectively combine encoded representations from multiple sections could be beneficial.

Summary updated

BibTeX
@inproceedings{frayling2024uog,
  author = {Erlend Frayling and Jake Lever and Graham McDonald},
  title = {{UoG Siephers at “Discharge Me!”: Exploring Ways to Generate Synthetic Patient Notes From Multi-Part Electronic Health Records}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure