Investigating the Utility of Synthetic Data for Doctor-Patient Conversation Summarization

Siyuan Chen, Colin A. Grambow, Mojtaba Kadkhodaie Elyaderani, Alireza Sadeghi, Federico Fancellu, Thomas Schaaf2023 · DOI

TL;DR

This paper explores using synthetic doctor-patient conversations to pre-train transformer models for generating clinical note summaries. They found that synthetic data can be as effective as real data for pre-training, offering a cost-effective way to improve summarization performance, especially when combined with other data sources.

What Problem Does This Paper Address?

Generating accurate and concise clinical notes from doctor-patient conversations is crucial for efficient healthcare but faces several challenges. Firstly, clinical datasets are often expensive, time-limited, and contain sensitive patient information, making large-scale pre-training difficult. Secondly, doctor-patient conversations (DoPaCos) can be very long, exceeding the input limits of many standard transformer models. While previous work has explored abstractive summarization and specialized models for long documents, this paper focuses on the utility of synthetic data for pre-training these summarization models, aiming to overcome data scarcity and cost issues.

Methods

The core of this work involves generating synthetic DoPaCos and using them to pre-train a Longformer-Encoder-Decoder (LED) model, a transformer architecture designed for long documents.

  1. Synthetic DoPaCo Generation: The authors fine-tuned a pre-trained DialoGPT model on 95% of their authentic, unlabeled DoPaCo dataset. This fine-tuned model was then used to generate approximately 160,000 synthetic DoPaCos. They experimented with different sampling strategies (top-k and top-p) to balance computational efficiency and the quality of generated conversations, ultimately choosing a combination that closely matched the word distribution of authentic data.
  2. Pre-training and Fine-tuning: The generated synthetic DoPaCos, along with a corpus of authentic clinical notes (SOAP notes), were used for in-domain pre-training of an LED model. They explored several pre-training strategies:
  3. Pre-training solely on synthetic DoPaCos.
  4. Pre-training solely on authentic clinical notes.
  5. Pre-training on a combination of synthetic DoPaCos and clinical notes.
  6. "Mix-and-match" experiments where the encoder and decoder weights were pre-trained separately on different data modalities (synthetic DoPaCos, clinical notes, or both).
  7. Finally, the pre-trained models were fine-tuned on a smaller, annotated dataset of DoPaCos with corresponding human-written HPI summaries.

Datasets

  • Authentic DoPaCos: Approximately 80,000 manually transcribed and de-identified doctor-patient conversations. This dataset lacked reference summaries, making it unsuitable for direct fine-tuning of summarization models.
  • Annotated DoPaCos: A separate set of 1,342 DoPaCos with corresponding human-written History of Present Illness (HPI) summaries. This was split into training (939), validation (201), and testing (202) sets.
  • Clinical Notes Corpus: A large corpus of approximately 475,000 proprietary, de-identified clinical notes (SOAP notes) used for pre-training.

Key Insights and Ideas

  • Synthetic Data as a Viable Pre-training Resource: The most significant finding is that synthetic DoPaCos can be a valuable substitute for authentic data in the pre-training phase. While not always outperforming authentic data, they consistently improve performance over a baseline with no in-domain pre-training. This suggests that synthetic data can effectively teach the model the nuances of the medical conversation domain without requiring access to costly and sensitive real-world data.
  • The Power of Modality Mixing: Separately pre-training the encoder and decoder on different data modalities (e.g., encoder on conversations, decoder on clinical notes) can yield superior results. This "mix-and-match" approach leverages the strengths of each data type: conversations for understanding the source, and clinical notes for learning the target summary format.
  • Conversational Data is Key for Summarization: Pre-training solely on clinical notes (for either encoder or decoder) generally led to worse performance compared to using conversational data. This highlights that exposure to the conversational structure and flow is crucial for models tasked with summarizing such interactions.

Key Results

  • Improved Summarization Metrics: Pre-training with either authentic or synthetic DoPaCos consistently improved ROUGE F1 scores and BERT scores compared to a baseline model without in-domain pre-training.
  • Synthetic vs. Authentic: Models pre-trained on synthetic DoPaCos achieved performance comparable to those pre-trained on authentic DoPaCos, outperforming the baseline in 6 out of 8 metrics.
  • "Mix-and-Match" Success: The best-performing model (Experiment No. 7 in Table 5) combined an encoder pre-trained on both synthetic DoPaCos and clinical notes with a decoder pre-trained solely on synthetic DoPaCos. This model achieved the highest ROUGE-1 score (39.34%) and BERT score (33.38%).
  • Qualitative Improvements: Human evaluations indicated perceivable improvements in the overall quality of generated summaries when models were pre-trained using DoPaCos.
  • Stability: Pre-training with DoPaCos also led to improved stability during the fine-tuning process.

Limitations and Future Work

  • Synthetic Data Quality: While effective, the paper acknowledges that synthetic data is assumed to be of lower quality than real conversations. The authors hypothesize that increasing quantity compensates for this.
  • Concept Extraction Nuances: While ROUGE and BERT scores improved, concept extraction metrics (UMLS and NER) showed mixed results, with some decreases for models pre-trained on synthetic data. This suggests that while overall fluency and semantic similarity improve, capturing specific medical concepts might still be a challenge.
  • Mechanism of "Mix-and-Match": The underlying reason why combining encoder and decoder weights pre-trained on different modalities works so well is not fully understood and warrants further investigation.
  • Future Directions: The authors suggest that this approach of using synthetic data for pre-training is inexpensive and viable when multiple datasets of distinctive natures are available, paving the way for more efficient model development in data-scarce domains.

Summary updated

BibTeX
@misc{chen2023investigating,
  author = {Siyuan Chen and Colin A. Grambow and Mojtaba Kadkhodaie Elyaderani and Alireza Sadeghi and Federico Fancellu and Thomas Schaaf},
  title = {{Investigating the Utility of Synthetic Data for Doctor-Patient Conversation Summarization}},
  year = {2023},
  doi = {10.21437/interspeech.2023-913},
}
Figure