Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images

Che Liu, Anand Shah, Wenjia Bai, Rossella ArcucciarXiv 2023 · OpenReview

Abstract

Medical Vision-Language Pre-training (VLP) typically relies on massive datasets of paired medical images and radiology reports. However, the logistical and ethical challenges of sharing real patient data often hinder progress. This research investigates whether VLP can be effectively implemented using entirely synthetic images generated from authentic radiology reports. By comparing general-purpose and domain-specific generative models, the study demonstrates that synthetic data can match or even exceed the performance of real images in downstream tasks like classification, segmentation, and object detection. We introduce a large-scale synthetic medical image dataset to facilitate privacy-preserving medical AI research.

The Challenge of Medical Data Sharing

Training robust medical AI models requires high-quality, paired image-text data. In the clinical domain, acquiring such data is fraught with difficulty. Privacy regulations like HIPAA and GDPR necessitate rigorous anonymization, which is both costly and time-consuming. Even when anonymized, medical images carry a residual risk of re-identification, making institutions hesitant to share large-scale datasets.

The central question of this research is whether we can bypass the need for real images entirely by using generative AI to transform clinical reports into high-fidelity synthetic images. If successful, this approach would allow for the distribution of large-scale pre-training datasets without the associated privacy risks of real patient pixels.

Methodology: Generating Medical Reality

The researchers developed a pipeline to replace real medical images with synthetic equivalents during the pre-training phase of VLP. They utilized the "Impression" section of over 213,000 radiology reports from the MIMIC-CXR collection as prompts for two different diffusion-based generative models:

  1. Stable Diffusion 2.1 (SD): A general-purpose model trained on natural images.
  2. RoentGen: A specialized model fine-tuned specifically for chest X-rays (CXRs).
Comparison of synthetic image generation quality between general-purpose Stable Diffusion and domain-specific RoentGen against real ground truth images.

As shown in the figure above, domain specificity is critical. While general-purpose models like Stable Diffusion produce images that vaguely resemble X-rays, they lack anatomical correctness, often hallucinating structures. In contrast, RoentGen produces medically plausible images that align closely with the clinical descriptions in the reports.

Performance Across Downstream Tasks

The models were pre-trained solely on synthetic image-real text pairs using three state-of-the-art VLP algorithms: ConVIRT, GLoRIA, and MGCA. These models were then evaluated on real-world datasets for classification (CheXpert, RSNA), segmentation (SIIM), and object detection.

Key Results:

  • Classification: On the CheXpert dataset, the GLoRIA framework pre-trained on RoentGen synthetic images achieved an AUC of 89.9, outperforming the model trained on real images (88.0).
  • Segmentation: MGCA trained on synthetic data achieved a 64.5 Dice score on the SIIM pneumothorax task, slightly exceeding the real-image baseline of 64.2.
  • Object Detection: RoentGen-based pre-training consistently improved mean Average Precision (mAP) across frameworks, with MGCA reaching 25.3 mAP compared to 24.9 for real images.

Discussion: The "Super-Augmentor" Effect

A surprising finding was that synthetic data often outperformed real data. The authors hypothesize that generative models may act as a "super-augmentor" or a feature-strengthening tool. By generating an image based strictly on the text report, the model may effectively "denoise" the data, focusing the image content on the specific clinical features described and removing irrelevant anatomical variations present in real patient scans.

Furthermore, models utilizing local or regional alignment (like GLoRIA and MGCA) saw the greatest benefits. This suggests that modern domain-specific generators are now capable of producing medically meaningful local features—such as specific opacities or nodules—that align accurately with textual descriptions.

Conclusion and Future Directions

This work demonstrates that synthetic data is a viable and potentially superior alternative to real images for medical VLP. By releasing a dataset of over 200,000 synthetic chest X-rays paired with real reports, the researchers provide a path forward for open-source medical AI development that respects patient privacy. Future work will explore the application of this method to 3D modalities like CT and MRI, as well as its utility in zero-shot learning and domain adaptation.

Summary updated

Abstract

Medical Vision-Language Pre-training (VLP) learns representations jointly from medical images and paired radiology reports. It typically requires large-scale paired image-text datasets to achieve effective pre-training for both the image encoder and text encoder. The advent of text-guided generative models raises a compelling question: Can VLP be implemented solely with synthetic images generated from genuine radiology reports, thereby mitigating the need for extensively pairing and curating image-text datasets? In this work, we scrutinize this very question by examining the feasibility and effectiveness of employing synthetic images for medical VLP. We replace real medical images with their synthetic equivalents, generated from authentic medical reports. Utilizing three state-of-the-art VLP algorithms, we exclusively train on these synthetic samples. Our empirical evaluation across three subsequent tasks, namely image classification, semantic segmentation and object detection, reveals that the performance achieved through synthetic data is on par with or even exceeds that obtained with real images. As a pioneering contribution to this domain, we introduce a large-scale synthetic medical image dataset, paired with anonymized real radiology reports. This alleviates the need of sharing medical images, which are not easy to curate and share in practice. The code and the dataset can be found in \href{https://github.com/cheliu-computation/MedSyn-RepLearn/tree/main}{https://github.com/cheliu-computation/MedSyn-RepLearn/tree/main}.
BibTeX
@misc{liu2023utilizing,
  author = {Che Liu and Anand Shah and Wenjia Bai and Rossella Arcucci},
  title = {{Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images}},
  year = {2023},
  eprint = {2310.07027},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.07027},
}
Figure