NewAgeHealthWarriors at MEDIQA-Chat 2023 Task A: Summarizing Short Medical Conversation with Transformers

Prakhar Mishra, Ravi Theja DesettyACL 2023

TL;DR

This paper tackles the challenge of automatically generating concise clinical notes from doctor-patient conversations, a crucial but time-consuming task in healthcare. The authors propose an ensemble approach combining rule-based systems, traditional sequence models, and advanced transformer models like BART and GPT-3, achieving competitive results on the MEDIQA-Chat 2023 benchmark.

What Problem Does This Paper Address?

The increasing adoption of telecare has highlighted the need for efficient ways to document patient interactions. Manually creating clinical notes after each doctor-patient encounter is a significant bottleneck, consuming valuable time and resources, especially when done at scale. This paper addresses the MEDIQA-Chat 2023 shared task, specifically Task A: Short Dialogue2Note Summarization, which aims to automatically generate a summary of a short medical conversation, including a relevant section header and its corresponding text. This automated process is vital for ensuring continuity of care, allowing medical professionals to quickly grasp patient history and for patients to refer back to their consultations.

Methods

The proposed system employs a two-pipeline approach: one for Section Header Identification and another for Section Text Generation.

For Section Header Identification, a two-step strategy is used. First, a "Keyword-based Categorizer" manually curates a list of keywords to map conversations to section headers. If this fails, a fine-tuned Bio_ClinicalBERT model is employed.

For Section Text Generation, the core of the system is an ensemble of transformer-based models. This ensemble includes:

  • BART-large and BioBART-large: These models are fine-tuned on the training dataset in a sequence-to-sequence paradigm.
  • Few-shot GPT-3: A prompt engineering approach is used, where GPT-3 is given examples of dialogues, section headers, and corresponding summaries to learn the desired output style. The authors experimented with different prompt structures and the number of examples, finding that 3 examples worked best.

The final summary is generated by scoring the outputs from individual models and selecting the best one based on a weighted combination of central and syntactic similarity scores.

Datasets

The primary dataset used is from the MEDIQA-Chat 2023 shared task. This dataset consists of doctor-patient conversations paired with corresponding section headers and texts. The paper mentions training, validation, and test sets with statistics provided in Table 1. The dataset is designed to foster research in automatic clinical note generation and summarization.

Key Insights and Ideas

  • Ensemble power: Combining diverse models (rule-based, traditional NLP, and large language models) proved more effective than any single model. This highlights the benefit of leveraging different strengths.
  • Few-shot GPT-3's potential: Even without extensive fine-tuning, GPT-3 in a few-shot setting demonstrated strong performance, outperforming fully supervised models in some instances. This underscores the growing capability of LLMs for specialized tasks with minimal adaptation.
  • Importance of prompt engineering: For GPT-3, carefully crafting prompts with relevant examples and context was crucial for achieving good results, demonstrating the art and science of interacting with LLMs.
  • Domain-specific models matter: The use of BioBART and Bio_ClinicalBERT, models pre-trained on biomedical data, provided an advantage over general-purpose models, emphasizing the value of domain adaptation.

Key Results

The proposed ensemble system achieved a score of 0.730 on the test set for Task A, ranking 8th on the leaderboard. This performance was better than a baseline system using only BART variants. Specifically, the ensemble of BART, BioBART, and GPT-3 outperformed individual models by approximately 0.9+ Rouge-Avg points on the validation dataset. The Section Header Identification pipeline achieved an accuracy of 77% on the validation set when incorporating the keyword-based categorizer.

Limitations and Future Work

The authors acknowledge that their model sometimes generated "made-up facts" or elaborated on details not present in the ground truth, impacting metrics like R1. They also note that while their model captured diagnostic details well, it sometimes categorized information in ways not present in the original ground truth.

For future work, the authors plan to:

  • Explore advanced pre-trained models and techniques for further performance improvements.
  • Investigate the applicability of their approach to more complex and longer dialogues.
  • Conduct an in-depth analysis of generated summaries to identify areas for fine-tuning.

Summary updated

BibTeX
@inproceedings{mishra2023newagehealthwarriors,
  author = {Prakhar Mishra and Ravi Theja Desetty},
  title = {{NewAgeHealthWarriors at MEDIQA-Chat 2023 Task A: Summarizing Short Medical Conversation with Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure