HealthMavericks@MEDIQA-Chat 2023: Benchmarking different Transformer based models for Clinical Dialogue Summarization

Kunal Suri, Saumajit Saha, Atul SinghACL 2023

TL;DR

This paper benchmarks various Transformer-based models for clinical dialogue summarization, a crucial but underexplored area. The authors found that fine-tuned Transformer models perform comparably to prompt-based large language models like GPT-3, offering a cost-effective and privacy-preserving alternative for clinical applications.

What Problem Does This Paper Address?

Manually summarizing doctor-patient conversations is time-consuming and expensive, hindering efficient record-keeping, training, and evaluation in healthcare. While Transformer models have shown promise in general dialogue summarization, their performance on the unique challenges of clinical dialogues, which require high accuracy and adherence to specific medical note structures, remains less understood. This paper addresses this gap by systematically benchmarking several state-of-the-art Transformer architectures on the MEDIQA-Chat 2023 clinical dialogue summarization task.

Methods

The authors tackle two subtasks:

  • SubTask A: Section Header Prediction and Dialogue Summarization: This involves predicting the relevant clinical note section header for a given dialogue and then generating a summary for that section. For section header prediction, they fine-tuned Bio-ClinicalBERT, employing Focal Loss to handle class imbalance. For summarization, they fine-tuned several Transformer Sequence-to-Sequence (Seq2Seq) models, including BioBart, Flan-T5, DialogLED, and also experimented with OpenAI GPT-3 using in-context learning. They explored two ensemble methods for combining model outputs: one based on semantic similarity and another minimizing hallucination.
  • SubTask B: Full Clinical Note Generation: This task requires generating a complete clinical note from a doctor-patient conversation. The approach involves a multi-label classifier to assign each utterance to a specific section, followed by Seq2Seq models to summarize the utterances within each section. Similar to SubTask A, they fine-tuned Transformer models and used OpenAI GPT-3.

A key aspect of their methodology is the use of Beam Search with TPESampler for optimizing decoding strategies to maximize ROUGE-1, ROUGE-2, and BertScore. They also employed early stopping based on validation metrics to select the best models.

Datasets

The primary dataset used is from the MEDIQA-Chat 2023 challenge.

  • SubTask A: Consists of <dialogue, section-text, section-header> triplets. The training data has 1201 triplets, and the validation data has 180.
  • SubTask B: Consists of <dialogue, note> pairs. The training data has 67 pairs, and the validation data has 20.

The paper does not introduce a new benchmark dataset but rather utilizes and analyzes performance on the existing MEDIQA-Chat 2023 dataset.

Key Insights and Ideas

  • Fine-tuned Transformers vs. Prompted LLMs: The study reveals that fine-tuned Transformer models (like DialogLED) can achieve performance comparable to prompt-based large language models (like OpenAI GPT-3) for clinical dialogue summarization. This is significant because fine-tuning offers potential advantages in terms of cost and data privacy, which are critical in healthcare.
  • Metric Suitability: The authors observed that ROUGE scores, while commonly used, might favor longer summaries and not fully capture the quality of summaries generated by models like GPT-3, which might be syntactically different but semantically accurate. Metrics like BertScore and BLEURT appear to be more suitable for evaluating such models due to their focus on semantic similarity.
  • Ensemble Effectiveness: Their experiments suggest that ensemble techniques can improve summary quality. The "Generating Best Summary by semantic similarity" approach showed promise, though the "minimizing hallucination" method, while aiming for faithfulness, sometimes underperformed on standard metrics, possibly due to the hallucination detection model not being trained on clinical data.
  • Model Performance: DialogLED-Large and DialogLED-Base models consistently outperformed other Transformer architectures tested, indicating their suitability for this domain.

Key Results

  • SubTask A: The authors' solution for SubTask B was ranked fifth among all submissions. While specific accuracy numbers for all models are detailed in the appendices, the paper highlights that their fine-tuned models performed well. For example, in Table 3, Run 3 (OpenAI GPT3) achieved a ROUGE-1 score of 0.5703, while Run 2 (semantic similarity ensemble) achieved 0.6268.
  • SubTask B: The paper reports results for different runs. Run 1 (minimizing hallucination ensemble) achieved the best ROUGE-1 score of 0.5311, but with more hallucinations. Run 2 (OpenAI GPT3) had fewer hallucinations but a lower score (0.2759). Run 3 (semantic similarity ensemble) performed better than Run 2 but still had hallucinations.

Limitations and Future Work

  • Hallucination Mitigation: The authors hypothesize that the "minimizing hallucination" ensemble approach might not have performed optimally because the hallucination detection model was not specifically trained on clinical data. This suggests a need for domain-specific hallucination detection.
  • Metric Limitations: The paper points out that ROUGE scores might not be the most appropriate metric for evaluating models like GPT-3, which can generate semantically accurate but syntactically different summaries. Future work could explore more robust semantic evaluation metrics.
  • OpenAI GPT-3 Cost: The authors mention that they couldn't fully analyze OpenAI GPT-3 due to cost constraints, limiting a comprehensive comparison.
  • Future Work: The paper suggests that identifying medical terms in generated summaries could be a future direction for evaluating effectiveness. Additionally, exploring more advanced ensemble techniques and domain-specific pre-trained models remains an open area.

Summary updated

BibTeX
@inproceedings{suri2023healthmavericks,
  author = {Kunal Suri and Saumajit Saha and Atul Singh},
  title = {{HealthMavericks@MEDIQA-Chat 2023: Benchmarking different Transformer based models for Clinical Dialogue Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure