DS4DH at MEDIQA-Chat 2023: Leveraging SVM and GPT-3 Prompt Engineering for Medical Dialogue Classification and Summarization

Boya Zhang, Rahul Mishra, Douglas TeodoroACL 2023

TL;DR

This paper tackles the challenge of information overload in healthcare by developing a two-stage system for classifying and summarizing medical dialogues. By combining a traditional SVM classifier with the advanced GPT-3.5 language model, the researchers achieved competitive results in the MEDIQA-Chat 2023 challenge, demonstrating a practical approach to improving clinical documentation efficiency.

What Problem Does This Paper Address?

Healthcare professionals are drowning in a sea of textual data, primarily from electronic health records (EHRs). This "information overload" hinders their ability to process patient information efficiently, reduces productivity, and can even lead to suboptimal medical decisions. Medical dialogues, in particular, are complex and time-consuming to digest. To combat this, clinical text summarization methods have been proposed to create concise representations of this data, helping clinicians focus on what's most important. However, accurately classifying the type of medical dialogue and then generating a high-quality summary remains a significant challenge. This paper addresses this by proposing a novel approach that integrates dialogue classification with advanced language models for summarization, aiming to streamline the process of understanding and documenting patient encounters.

Methods

The researchers employed a two-stage pipeline for their approach.

  1. Short Dialogue Classification: They utilized a Support Vector Machine (SVM) classifier, a well-established machine learning technique, to categorize short medical dialogues. The text was first transformed into a token count matrix, then converted to a TF-IDF representation. An SGD optimization algorithm with L2 penalty was used, and the classifier was calibrated to provide probability estimates. This stage aims to identify the specific section header (e.g., "Past Medical History," "Diagnosis") that the dialogue pertains to.
  2. Short Dialogue Summarization: This stage leveraged OpenAI's GPT-3.5 model. Two main strategies were explored:
  3. Run 1 (One-Shot Prompting): GPT-3.5 was used with a "one-shot" prompting strategy. For each test dialogue, a random training instance with the same predicted section header was selected. This training instance (dialogue and its summary) was then used as context for GPT-3.5 to generate a summary for the new test dialogue.
  4. Run 2 (Fine-tuning): A smaller GPT-3 model (Curie) was fine-tuned on the entire training dataset. Summaries were then generated using this fine-tuned model.

The key innovation here is the combination of a classical classifier with a powerful generative model, and specifically, the exploration of prompt engineering techniques (using same-header context) for GPT-3.5 to improve summarization quality.

Datasets

The study used the MTS-Dialog dataset, a collection of medical dialogues from doctor-patient encounters. This dataset comprised 1201 training instances, 100 validation instances, and 200 test instances for the competition. Each instance included an identifier, section header, dialogue, and summary. The paper notes that the dataset was used for both training and evaluation, and it was part of the MEDIQA-Chat 2023 Task A (Dialogue2Note Summarization).

Key Insights and Ideas

  • The Power of Contextual Prompting: The study found that using a training dialogue with the same predicted section header as context for GPT-3.5 (Run 1) significantly improved summarization performance compared to using a random training instance. This highlights the importance of providing relevant context to large language models for domain-specific tasks.
  • SVM's Continued Relevance: Despite the rise of deep learning, the SVM classifier proved effective and computationally efficient for the initial dialogue classification task, demonstrating that traditional methods still hold value.
  • GPT-3.5's Adaptability: The GPT-3.5 model, even with a one-shot prompt strategy, showed strong performance, suggesting its ability to adapt to medical dialogue summarization with minimal task-specific training data.
  • Classification as a Precursor: The two-stage approach, where classification precedes summarization, is crucial. Accurately identifying the dialogue's category allows for more targeted and relevant summarization.

Key Results

  • Classification: The SVM model achieved an accuracy of 0.70 on the test set, surpassing the average participant's accuracy (0.56) but falling slightly below the best participant (0.78).
  • Summarization:
    • Run 1 (GPT-3.5 with same-header prompt): Achieved an aggregate score of 0.4977, outperforming the average participant's aggregate score by 2.4 percentage points. This run yielded better results than Run 2 across all metrics (ROUGE-1, BERTScore F1, BLEURT).
  • Run 2 (Fine-tuned GPT-3 Curie): Achieved an aggregate score of 0.4334.
  • The best model (Run 1) was outperformed by the top-ranked run by 8 percentage points, similar to the classification results.
  • Qualitative Analysis: The Run 1 summary was described as more coherent and clear than the Run 2 summary, effectively conveying key patient information. The Golden summary was the most comprehensive, but Run 1 provided a strong, concise account.

Limitations and Future Work

  • Limitations:
    • The classification performance, while competitive, was not top-tier, indicating room for improvement.
  • The summarization performance, though good, was still outperformed by the challenge's best run.
  • The study acknowledges that automated evaluation metrics (ROUGE, BERTScore, BLEURT) may not fully capture human judgment of summary quality.
  • Future Work:
    • Classification: Explore alternative machine learning algorithms or fine-tune language models specifically for medical dialogue classification. Incorporating domain-specific knowledge (e.g., medical ontologies) could also enhance accuracy.
  • Summarization: Refine prompt strategies, experiment with different configurations, and incorporate more context. Investigating chain-of-thought reasoning could lead to more coherent summaries.
  • Evaluation: Conduct user studies with medical professionals to assess the practical utility and accuracy of the generated summaries in real-world clinical settings.
  • Explore multi-task learning or domain-specific fine-tuning for improved performance.

Summary updated

BibTeX
@inproceedings{zhang2023ds,
  author = {Boya Zhang and Rahul Mishra and Douglas Teodoro},
  title = {{DS4DH at MEDIQA-Chat 2023: Leveraging SVM and GPT-3 Prompt Engineering for Medical Dialogue Classification and Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure