Team Cadence at MEDIQA-Chat 2023: Generating, augmenting and summarizing clinical dialogue with large language models

Ashwyn Sharma, David Feldman, Aneesh JainACL 2023

TL;DR

This paper details Team Cadence's winning approach to the MEDIQA-Chat 2023 challenge, focusing on summarizing and generating clinical dialogues. They achieved top performance by leveraging the BART model, employing a novel "N-pass" strategy for long conversations, and significantly boosting performance through data augmentation derived from a dialogue generation task.

What Problem Does This Paper Address?

The paper tackles the challenge of processing and generating clinical dialogues, a complex and crucial area in healthcare. Specifically, it addresses three tasks within the MEDIQA-Chat 2023 challenge:

  • Task A (Short Dialogue2Note Summarization): Generating a section summary (header and content) from a short clinical conversation.
  • Task B (Full Dialogue2Note Summarization): Creating a complete clinical note from an entire patient-doctor dialogue.
  • Task C (Note2Dialogue Generation): Generating realistic patient-doctor conversations from input clinical notes, serving as a data augmentation task.

The core difficulties lie in the abstractive nature of summarization, the need to capture key medical concepts accurately, and the challenge of handling long input sequences that often exceed the token limits of large language models. The limited availability of training data for these specialized tasks further exacerbates these issues.

Methods

Team Cadence's approach is built around the BART (Bidirectional and Auto-Regressive Transformer) architecture, chosen for its versatility and strong performance on sequence-to-sequence tasks.

  • Dialogue Generation (Task C): They ingeniously repurposed the SAMSum dataset, designed for dialogue summarization, for dialogue generation by inverting the input and target labels. This allowed them to generate synthetic patient-doctor conversations.
  • Data Augmentation: The generated synthetic dialogues were combined with the original training datasets for Tasks A and B. Additionally, they sampled notes from the MIMIC-IV-Note dataset to create more training data for Task C. This data augmentation strategy proved critical for improving performance.
  • N-pass Summarization (Task B): To overcome BART's token limit for long conversations in Task B, they developed a novel "N-pass" strategy. This method iteratively summarizes chunks of the dialogue, concatenating the previous summary with the next dialogue segment in each pass. This allows the model to process longer inputs by breaking them down into manageable steps.
  • Classification (Task A): For predicting section headers in Task A, they fine-tuned the BART model using the BartForSequenceClassification wrapper, adding a classification head on top of the BART encoder.

Datasets

  • MEDIQA-Chat 2023: The primary dataset for the shared tasks, providing training, validation, and test sets for Tasks A, B, and C.
  • SAMSum: A dialogue summarization dataset used for pre-training and for generating synthetic dialogues by inverting its input/output structure.
  • MIMIC-IV-Note: A large dataset of deidentified clinical notes, used to generate additional synthetic dialogues for data augmentation.

Key Insights and Ideas

  • Data Augmentation is King: The most significant takeaway is the profound impact of data augmentation, particularly by generating synthetic clinical dialogues. This strategy dramatically improved performance across tasks, especially when the original datasets were small.
  • Task C as a Data Engine: Leveraging Task C (dialogue generation) as a tool to augment data for Tasks A and B was a highly effective strategy, demonstrating the interconnectedness of these tasks.
  • BART's Versatility: The BART model proved to be a robust and adaptable backbone, performing well across summarization, generation, and classification tasks with minimal architectural changes.
  • N-pass for Long Sequences: The N-pass summarization strategy is a novel and effective solution for handling the inherent token limitations of LLMs when dealing with very long documents, a common issue in clinical text.

Key Results

  • Task C: Team Cadence achieved rank-1 among all participants, significantly outperforming the next best submission.
  • Task B: The N-pass strategy improved ROUGE-2 scores by 8.1% on the validation set. The final submission (run2, using N-pass) outperformed the baseline run1 by 6.8% on the test set.
  • Task A: Data augmentation led to improvements of 13.1% and 14% in ROUGE-1 and ROUGE-2 scores, respectively, over the baseline.
  • Classification (Task A): Achieved 78% accuracy and 78.37% F1 score on the validation set.
  • Reproducibility: Team Cadence was noted as the only team to submit stable and reproducible runs for all three tasks.

Limitations and Future Work

  • External Medical Knowledge: The models did not leverage external medical knowledge, a technique that has shown promise in other studies.
  • Hallucinations and Omissions: Like other LLM-based approaches, the models are prone to generating hallucinations and omitting key clinical concepts. The authors plan to explore constrained beam search for mitigation.
  • Synthetic Dialogue Quality: While effective, the generated dialogues sometimes had too few turns or overly long individual turns. Future work could focus on generating more natural-flowing conversations.
  • N-pass Optimization: The N-pass strategy for Task B could be further explored with higher values of N to potentially improve coverage, though diminishing returns are expected.

Summary updated

BibTeX
@inproceedings{sharma2023team,
  author = {Ashwyn Sharma and David Feldman and Aneesh Jain},
  title = {{Team Cadence at MEDIQA-Chat 2023: Generating, augmenting and summarizing clinical dialogue with large language models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure