Team Converge at ProbSum 2023: Abstractive Text Summarization of Patient Progress Notes

Gaurav Kolhatkar, Aditya Paranjape, Omkar Gokhale, Dipali KadamACL 2023

TL;DR

This paper tackles the challenge of automatically summarizing patient progress notes to combat information overload for medical professionals. The team explored state-of-the-art abstractive summarization models, finding that Pegasus, when fine-tuned on the "Assessment" section of patient notes, achieved the best performance, ranking third on the ProbSum 2023 leaderboard.

What Problem Does This Paper Address?

The digitization of healthcare has led to an explosion of patient data, making it increasingly difficult for medical professionals to quickly sift through lengthy progress notes. This information overload can lead to missed crucial details and increased cognitive burden. The paper addresses the need for efficient and accurate automatic text summarization, specifically for clinical medicine, to help doctors and nurses extract the most important information from patient records, thereby improving efficiency and potentially reducing diagnostic errors. The goal is to create concise summaries that capture the essence of a patient's condition and progress.

Methods

The authors employed abstractive text summarization, which generates new sentences to capture the core meaning, as opposed to extractive summarization that simply selects existing sentences. They experimented with four prominent transformer-based models: Pegasus, BART, T5-small, and T5-base. These models were fine-tuned on a dataset of patient progress notes.

Key methodological aspects include:

  • Feature Selection: Investigating the impact of different input features (Assessment, Objective, Subjective sections of notes) on summarization performance.
  • Preprocessing: Applying techniques like stop-word removal and lemmatization to clean the input text.
  • Data Augmentation: Attempting to improve performance by augmenting the training data with a different dataset (MeQSUM).
  • Model Architectures: Leveraging pre-trained models from HuggingFace, including Pegasus ("google/pegasus-large"), BART ("facebook/bart-large"), and T5 ("t5-small" and "t5-base").

Datasets

The primary dataset used for training and evaluation was sourced from MIMIC-III, a publicly available de-identified electronic health record dataset. For this specific task, the dataset consisted of approximately 60,000 hospital ICU admissions. The training set contained 765 samples, and the test set had 237 samples. The notes were structured with features like FILE ID, Subjective Sections, Objective Sections, Assessment, and Summary (ground truth).

Additionally, the authors experimented with augmenting their training data using the MeQSUM dataset, which contains 1000 consumer health questions and their summaries.

Key Insights and Ideas

  • The "Assessment" Section is Key: The most significant finding was that using only the "Assessment" section of the patient notes yielded the best summarization results. This suggests that this specific part of the progress note contains the most critical information for generating a concise summary of a patient's status.
  • Abstractive Summarization's Advantage: The paper highlights that abstractive summarization is inherently more flexible than extractive methods for this task. While extractive methods are limited to the original text, abstractive models can generate novel phrasing, potentially leading to summaries that more closely match the ground truth.
  • Preprocessing Offers Modest Gains: Basic preprocessing techniques like stop-word removal and lemmatization showed a slight improvement in model performance, indicating that cleaning the input text can be beneficial.
  • Data Augmentation Challenges: Augmenting the training data with the MeQSUM dataset did not significantly improve performance. The authors hypothesize this is due to structural differences between the MeQSUM dataset (question-answer format) and the patient notes dataset.

Key Results

The paper reports that Pegasus was the best-performing model on the test dataset, achieving a ROUGE-L F1 score of 0.2744. This performance placed them third on the ProbSum 2023 leaderboard.

Specific results from their experiments include:

  • Feature Importance: Models trained solely on the "Assessment" feature achieved the highest average ROUGE-L F1 score (0.2302), outperforming combinations with "Objective" and "Subjective" sections.
  • Preprocessing Impact: Table 2 shows that preprocessing generally led to slight improvements across models, for example, Pegasus's score increased from 0.2744 (Raw Dataset) to 0.2308 (Preprocessing).
  • Data Augmentation Impact: Augmenting with MeQSUM did not yield significant improvements, with Pegasus scoring 0.2307 with augmentation compared to 0.2744 without.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Data Augmentation Mismatch: The MeQSUM dataset's structural differences from the task dataset hindered the effectiveness of data augmentation.
  • Limited Model Ensemble: The experiments focused on individual models rather than exploring ensemble methods.

Future work suggested by the authors includes:

  • Similar Data Augmentation: Augmenting the data with datasets that have a structure more similar to the patient progress notes.
  • Ensemble Models: Implementing an ensemble of the summarization models used in this study to potentially achieve even better performance.

Summary updated

BibTeX
@inproceedings{kolhatkar2023team,
  author = {Gaurav Kolhatkar and Aditya Paranjape and Omkar Gokhale and Dipali Kadam},
  title = {{Team Converge at ProbSum 2023: Abstractive Text Summarization of Patient Progress Notes}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure