Generation and Evaluation of Synthetic Endoscopy Free-Text Reports with Differential Privacy

Agathe Zecevic, Xinyue Zhang, Sebastian Zeki, Angus RobertsACL 2024

TL;DR

This paper tackles the challenge of training robust AI models for medical text analysis, which is hindered by the scarcity of patient data due to privacy concerns. The authors propose a novel approach using a fine-tuned language model (BioGPT) with Differential Privacy (DP) to generate synthetic endoscopy reports. Their findings suggest that DP-generated synthetic data can improve downstream AI task performance while offering a promising balance between data utility and patient privacy.

What Problem Does This Paper Address?

Developing advanced AI tools in healthcare, particularly those relying on Natural Language Processing (NLP), is critically dependent on large, high-quality datasets of patient information. However, the availability of such data is severely restricted by privacy regulations, ethical considerations, and the sheer difficulty of accessing and sharing sensitive medical records. This scarcity can lead to AI models that are biased, perform poorly on unseen data, and are not generalizable. While synthetic data offers a potential solution, ensuring that this synthetic data truly preserves patient privacy is paramount. Simply removing direct identifiers is often insufficient, and existing methods for generating synthetic medical text with robust privacy guarantees are limited, especially for specialized domains like gastroenterology endoscopy reports.

Methods

The core of this research involves fine-tuning a pre-trained language model, BioGPT, on a large dataset of real endoscopy reports. The key innovation lies in integrating Differential Privacy (DP) into the training process. DP is a mathematical framework that guarantees a certain level of privacy by adding noise during training, making it statistically difficult to determine if any specific individual's data was included in the training set. Specifically, the authors employed Differentially Private Stochastic Gradient Descent (DP-SGD), which involves clipping gradient norms and adding Gaussian noise.

For generating synthetic reports, they used a text generation pipeline that was iteratively refined with clinician feedback. They controlled the generation process using "control codes" to specify report types (e.g., OGD, colonoscopy) and set parameters like maximum length, temperature (to balance creativity and accuracy), and "no repeat ngram size" to ensure report quality and uniqueness.

To evaluate the generated reports, they employed a multi-faceted approach:

  • Text Similarity: Using ROUGE-L scores to measure how closely synthetic reports resemble original reports, comparing DP-generated reports to those generated without DP.
  • Language Quality: Employing a language scoring model to assess the fluency and coherence of the synthetic text.
  • Utility: Evaluating the usefulness of the synthetic data in downstream NLP tasks, specifically a 4-class classification problem related to Barrett's Esophagus detection. This involved using synthetic data for data augmentation and for task-adaptive pre-training (TAPT).

Datasets

The primary dataset used for training the BioGPT model and generating synthetic reports consisted of 93,162 real endoscopy reports from St Thomas' Hospital in London, spanning January 2017 to October 2023. These reports covered various procedures including Colonoscopy, Gastroscopy, Endoscopic Ultrasound (EUS), Sigmoidoscopy, and ERCP. For evaluation purposes, a separate set of 10,000 real endoscopy reports (not used in training) was also utilized, matched for variety and number to the synthetic dataset.

Key Insights and Ideas

  • DP as a Privacy Enabler: The study demonstrates that Differential Privacy can effectively reduce the similarity between synthetic reports and the original training data, as evidenced by a significant shift in ROUGE-L score distributions for DP-generated reports compared to non-DP generated ones. This suggests a stronger privacy guarantee.
  • The Privacy-Utility Trade-off is Real: While DP enhances privacy, it does lead to a slight reduction in language quality and a less pronounced improvement in downstream task performance compared to using real data. The paper highlights that a lower epsilon (stronger privacy) generally reduces utility.
  • Task-Adaptive Pre-training (TAPT) Shines: TAPT, especially when using synthetic data, proved to be a highly effective method for improving model performance on downstream tasks. This is particularly significant because TAPT doesn't require additional labeled data, addressing a major bottleneck in medical AI development.
  • Synthetic Data Can Augment Real Data: Even with the privacy-induced limitations, DP-generated synthetic data still showed benefits when used for data augmentation and TAPT, improving classification performance, particularly for underrepresented classes.
  • Outlier Protection: The authors specifically included unique clinical findings ("outliers") in their training set and confirmed that these were not directly regenerated in the synthetic reports, indicating a good level of protection against memorization of sensitive cases.

Key Results

  • Text Similarity: DP-generated reports showed significantly lower ROUGE-L scores (average 0.226) compared to non-DP generated reports (average 0.660), indicating greater divergence from the training set.
  • Language Quality: Reports generated with DP exhibited slightly lower language quality scores, with a broader, shorter-tailed distribution compared to non-DP generated reports.
  • Utility (Classification Task):
    • Task-adaptive pre-training (TAPT) with real patient data yielded the best performance across all classes.
  • TAPT with DP synthetic data significantly improved performance, especially for the 'Insufficient' class (F1-score increase of 0.034).
  • Data augmentation with DP synthetic reports also showed improvements, though with more variability.

Limitations and Future Work

The authors acknowledge several limitations:

  • Fixed Privacy Budget: The study used a single epsilon value (ε=4). Future work should explore the impact of varying epsilon on the privacy-utility trade-off.
  • Limited Annotation for Augmentation: The data augmentation experiments were constrained by the limited availability of manually annotated synthetic reports.
  • Population Bias: The data originated from a single London hospital, which might introduce population bias.
  • Clinical Relevance Not Evaluated: While text quality was assessed, the clinical accuracy and relevance of the generated reports were not formally evaluated, which is crucial for real-world medical applications.

Future work should focus on a more comprehensive evaluation of the privacy-utility trade-off, comparing their approach with other synthetic data generation methods, and investigating the clinical relevance of the generated reports.

Summary updated

BibTeX
@inproceedings{zecevic2024generation,
  author = {Agathe Zecevic and Xinyue Zhang and Sebastian Zeki and Angus Roberts},
  title = {{Generation and Evaluation of Synthetic Endoscopy Free-Text Reports with Differential Privacy}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure