IUTEAM1 at MEDIQA-Chat 2023: Is simple fine tuning effective for multi layer summarization of clinical conversations?

Dhananjay SrivastavaACL 2023

TL;DR

This paper investigates effective strategies for summarizing lengthy clinical conversations into concise "chart notes." The authors found that ensembling specialized summarization models for different sections of a chart note significantly improves performance, outperforming a single model. However, a multi-layer approach where a final model synthesizes these section-wise summaries did not yield further improvements.

What Problem Does This Paper Address?

The increasing adoption of Electronic Health Records (EHRs) has led to a significant burden on healthcare professionals, who spend a considerable amount of time documenting patient encounters. This documentation process is time-consuming and can contribute to physician burnout. Furthermore, the sheer volume of information generated during a patient visit makes it challenging for physicians to quickly identify and interpret the most critical details. To alleviate this, AI-powered summarization models are being developed to automatically extract and condense key information from clinical conversations into structured "chart notes." However, a major challenge in this domain is the limited availability of training data due to privacy concerns and the specialized, often varied, medical terminology used by healthcare professionals. This paper aims to explore effective ensembling strategies for multi-layer summarization of clinical conversations to improve the accuracy and coherence of generated chart notes.

Methods

The researchers explored three distinct approaches for summarizing clinical conversations using the LSG BART model, a transformer-based architecture capable of handling long sequences.

  1. Single LSG BART Model: A baseline approach using a single LSG BART model to summarize the entire conversation. This was also tested with and without fine-tuning on medical research papers from PubMed.
  2. Section-Wise Ensemble: This approach involves segmenting the input conversation into distinct sections (e.g., Chief Complaint, History of Present Illness, Physical Exam). A separate LSG BART model is trained for each section, and their outputs are concatenated to form the final chart note. This leverages the idea that text within a specific section is more coherent.
  3. Multi-Layer Ensemble: Building upon the section-wise ensemble, this method takes the concatenated summaries from Approach 2 and passes them through another LSG BART model. The goal is to create a more coherent and comprehensive final summary by allowing a final model to synthesize the section-wise outputs.

Datasets

The primary dataset used in this study is from the MEDIQA-Chat 2023 challenge, specifically Task B, which focuses on generating full chart notes from conversations. The dataset consists of training and validation sets with 67 and 20 conversations, respectively, and an additional hidden test set of 40 conversations. The evaluation was performed using ROUGE, BLEURT, and BERTScore metrics.

Key Insights and Ideas

  • Section Specialization Matters: The most significant finding is that tailoring summarization models to specific sections of a clinical note (Approach 2) yields better results than a single, general-purpose model (Approach 1). This suggests that the distinct nature and content of different chart note sections benefit from specialized processing.
  • Cascading Summaries Doesn't Always Help: Surprisingly, a multi-layer approach (Approach 3), where a final model synthesizes section-wise summaries, did not improve performance over the section-wise ensemble. This indicates that simply combining outputs from specialized models doesn't automatically lead to better overall coherence or accuracy.
  • PubMed Fine-tuning is Not a Silver Bullet: Fine-tuning models on medical literature from PubMed did not improve performance. The authors hypothesize this is because PubMed data is literature-based, not conversational, and the embeddings might not be optimized for conversational summarization.

Key Results

The study evaluated the models using ROUGE scores on the validation and hidden test sets.

  • Validation Set:
    • Single BART: RougeL 0.264
    • Section Wise: RougeL 0.305
    • Multi-Layer: RougeL 0.231
  • Hidden Test Set (Full Chart Note):
    • Single BART: RougeL 0.2545
    • Section Wise: RougeL 0.306
    • Multi-Layer: RougeL 0.2204
  • Hidden Test Set (Section-wise):
    • Single BART: Average RougeL 0.3525
    • Section Wise: Average RougeL 0.5351
    • Multi-Layer: Average RougeL 0.3848

These results clearly show that the Section-Wise Ensemble (Approach 2) consistently outperformed both the single model and the multi-layer approach across various metrics and evaluation types. In the competition, Approach 2 secured 7th place for full note generation and 5th for section-wise generation among 23 submitted models.

Limitations and Future Work

The authors acknowledge that language models can sometimes produce hallucinations, emphasizing that these systems should always be used with human supervision in a clinical setting. They also note that their study is an experiment and further research with larger datasets and more rigorous statistical analyses is needed.

Future work includes:

  • Performing in-depth analyses of model architectures for section-wise next-generation summarization.
  • Investigating other models used in Long Document Summarization to further improve overall accuracy.
  • Exploring model explainability studies to build trust and understanding in clinical applications.

Summary updated

BibTeX
@inproceedings{srivastava2023iuteam,
  author = {Dhananjay Srivastava},
  title = {{IUTEAM1 at MEDIQA-Chat 2023: Is simple fine tuning effective for multi layer summarization of clinical conversations?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure