MLBMIKABR at “Discharge Me!”: Concept Based Clinical Text Description Generation

Abir Naskar, Jane Hocking, Patty Chondros, Douglas Boyle, Mike ConwayACL 2024

TL;DR

This paper introduces "Concept Based Description Generation," a novel method for automatically creating structured summaries of hospital course and discharge instructions from clinical notes. By segmenting text, identifying key concepts, and fine-tuning Large Language Models (LLMs), the approach aims to improve the accuracy and informativeness of medical documentation.

What Problem Does This Paper Address?

Generating accurate and comprehensive discharge summaries is a critical but time-consuming task in healthcare. The "Discharge Me!" task, part of the BioNLP workshop, aims to automate the creation of "Brief Hospital Course" and "Discharge Instructions" sections within discharge notes. These sections are crucial for patient care continuity but are often lengthy and complex, requiring significant manual effort to synthesize information from various sources like discharge and radiology reports. Existing methods often struggle with the sheer volume of text and the need to extract specific, actionable information in a structured format.

Methods

The proposed "Concept Based Description Generation" method involves a multi-stage pipeline:

  1. Source Document Segmentation: Both discharge and radiology texts are segmented using rule-based approaches tailored to their specific formatting patterns (e.g., multiple newlines, specific characters like colons or asterisks). Noise removal is a crucial initial step.
  2. Target Document Segmentation and Concept Extraction: Target texts ("Discharge Instructions" and "Brief Hospital Course") are also segmented. Crucially, this stage involves identifying "concepts" within each segment. Concepts are defined as key topics like patient instructions, medications, or disease information. This is achieved by training LLMs to predict concepts based on text patterns and keywords.
  3. Summarization: To manage document length and prevent information loss, a summarization model (BioMistral-7B) is applied to individual text segments rather than the entire document. This is done using specific prompts designed to extract essential information concisely.
  4. Concept Generation and Description Generation: LLMs are then fine-tuned for two primary tasks:
  5. Concept Generation: Given the summarized source text, the LLM identifies potential concepts relevant to the "Discharge Instruction" or "Brief Hospital Course" task.
  6. Description Generation: For each identified concept, another fine-tuned LLM generates a specific description based on the summarized source text and the list of all extracted concepts. This ensures that the generated descriptions are contextually relevant and structured.

The overall approach emphasizes a structured generation process, moving from raw text to segmented information, then to identified concepts, and finally to detailed, concept-driven descriptions.

Datasets

The primary datasets used are derived from the MIMIC-IV-Note and MIMIC-IV-ED datasets, specifically for the "Discharge Me!" task. The competition provides defined training, validation, and testing sets comprising 109,168 emergency department admissions. The authors utilized a subset of 10,000 samples for training due to time constraints.

Key Insights and Ideas

  • Concept-driven summarization is key: The core innovation lies in moving beyond simple text summarization to a concept-driven approach. By identifying and generating descriptions for specific concepts, the output becomes more structured, relevant, and actionable for medical professionals.
  • Segmenting for efficiency and accuracy: Applying summarization at the segment level, rather than on the entire document, is a pragmatic approach to handle the extremely long clinical texts and mitigate the risk of losing critical details.
  • Rule-based segmentation as a foundation: While LLMs are used for concept extraction and description generation, the initial segmentation relies on robust rule-based methods. This provides a structured input for the LLMs, potentially improving their performance.
  • Fine-tuning LLMs for specific tasks: The paper highlights the effectiveness of fine-tuning LLMs for distinct sub-tasks (concept extraction and description generation) to achieve specialized and accurate outputs.

Key Results

The paper presents performance metrics in Table 1, showing the overall performance of their proposed model on the Phase-2 test data. The overall performance metric is 0.17, with individual metrics like BLEU at 0.04, ROUGE-1 at 0.21, ROUGE-2 at 0.1, ROUGE-L at 0.13, BERTScore at 0.18, Meteor at 0.30, and AlignScore at 0.20. These metrics are calculated using the task organizers' evaluation script on Codabench.

Limitations and Future Work

The authors acknowledge several limitations:

  • Segment-level summarization necessity: The need to summarize each text segment individually is a challenge, and the current rule-based methods have limitations.
  • Handling long segments: There are threshold limits for handling very long segments.
  • Model and prompt dependence: The quality of the generated output is dependent on the chosen LLM and the prompts used.
  • Time constraints: Limited time prevented comprehensive comparisons of different models and prompts.

Future work includes:

  • Developing new models for better segmenting lengthy documents.
  • Comparing different models and prompts to optimize performance.
  • Exploring advanced data segmentation techniques to improve accuracy and efficiency.
  • Addressing the impact of poor summarization on concept generation quality.

Summary updated

BibTeX
@inproceedings{naskar2024mlbmikabr,
  author = {Abir Naskar and Jane Hocking and Patty Chondros and Douglas Boyle and Mike Conway},
  title = {{MLBMIKABR at “Discharge Me!”: Concept Based Clinical Text Description Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure