MediSyn: Text-Guided Diffusion Models for Broad Medical 2D and 3D Image Synthesis

Joseph Cho, Cyril Zakka, Rohan Shad, Ross Wightman, Akshay Chaudhari, William Hiesinger

TL;DR

This paper introduces MediSyn, a generalist text-guided diffusion model capable of generating synthetic medical images across multiple specialties and modalities. By training on a diverse, publicly available dataset, MediSyn offers a computationally efficient and reproducible solution to the data scarcity problem in medical AI research, while also demonstrating its potential to improve classifier performance in data-limited scenarios.

What Problem Does This Paper Address?

Developing robust deep learning (DL) models in medicine is hampered by a critical challenge: data scarcity. This scarcity stems from strict patient privacy regulations, the difficulty of sharing sensitive medical data, and the sheer cost and effort involved in curating large, diverse datasets. While synthetic data generation offers a promising solution, existing image generative models are often task-specific, meaning they are trained for a single medical specialty or imaging modality. This limits their scalability and reproducibility, as a new model must be developed and maintained for each specific use case. Furthermore, many of these models rely on private datasets, hindering independent research and development. The authors aim to address this by creating a single, generalist model that can generate high-quality synthetic medical images across a broad range of applications.

Methods

The core of MediSyn is a text-conditional latent diffusion model (LDM), specifically fine-tuned from Stable Diffusion version 1.4. LDMs work by progressively denoising a latent representation of an image, guided by text prompts. The authors fine-tuned this model on a massive, publicly available dataset of medical image-text pairs.

Key aspects of their approach include:

  • Generalist Architecture: Instead of training separate models for each specialty or modality, MediSyn uses a single, unified U-Net architecture. This allows it to handle diverse text prompts and generate images across different medical domains.
  • Text Conditioning: The model leverages text prompts to guide the image generation process, enabling precise control over the type of medical image produced.
  • Publicly Available Data: A significant contribution is the exclusive use of publicly available data for training, enhancing reproducibility and accessibility.
  • Computational Efficiency: The generalist approach is designed to be more computationally efficient than managing a suite of task-specific models, particularly during inference.

Datasets

The primary dataset used for training MediSyn is a curated corpus of 1,260,826 medical image-text pairs. This dataset spans:

  • 6 Medical Specialties: Gastroenterology, Radiology, Pathology, Surgery, Dermatology, and Ophthalmology.
  • 10 Imaging Modalities: Computed Tomography, X-ray, Magnetic Resonance Imaging, Ultrasound, Endoscopy, Microscopy, Fundoscopy, Optical Coherence Tomography, Dermoscopy, and Clinical images.

The data was split into training (80%), validation (10%), and testing (10%) sets. For evaluation, the authors also used specific subsets of this data and created task-specific datasets for comparisons.

Key Insights and Ideas

  • Generalization Doesn't Hurt Quality: A key finding is that training a generative model on a visually diverse set of medical images does not degrade the quality of the synthetic images. This challenges the intuition that specialization might be necessary for high fidelity.
  • Efficiency Through Generality: A single generalist model is significantly more computationally efficient for generating diverse medical images than managing a collection of specialized models. This is primarily due to avoiding the overhead of loading and switching between numerous task-specific models.
  • Physician Validation: Expert physicians found it difficult to reliably distinguish between real and synthetic medical images, suggesting a high degree of realism and visual fidelity in MediSyn's outputs.
  • Privacy Assurance: The synthetic images generated by MediSyn were found to be visually distinct from their corresponding real patient images, alleviating concerns about data memorization and privacy breaches.
  • Boosting Classifier Performance: In data-limited scenarios, supplementing real training data with MediSyn-generated images significantly improved the performance of downstream classification models.

Key Results

  • Computational Efficiency: MediSyn generated 10,000 images in approximately 4.34 hours, which is more than twice as fast as a comparable suite of task-specific models (9.18 hours).
  • Image Quality: MediSyn achieved comparable or better FID (fidelity) and MS-SSIM (diversity) scores compared to task-specific models for dermoscopy and robot-assisted radical prostatectomy images. For musculoskeletal X-ray, it showed a slight trade-off, with higher FID but lower MS-SSIM.
  • Physician Assessment: Surgeons and ophthalmologists struggled to identify synthetic images, with low recall rates (around 43-50%) and modest precision (around 62-63%), indicating high realism.
  • Data Reproduction: Normalized Euclidean distances between synthetic and real images were generally high (e.g., 0.34 ± 0.07 for Radiology), with a low percentage of pairs falling below a threshold indicative of direct reproduction (e.g., < 0.15).
  • Classifier Improvement: In low-data settings (e.g., ≤ 25% of training data), classifiers trained with MediSyn-generated data (either alone or supplemented with real data) outperformed those trained solely on real data across multiple medical specialties.

Limitations and Future Work

The authors acknowledge several limitations:

  • Lack of Zero-Shot Capability: MediSyn cannot generate images for modalities it hasn't been explicitly trained on (e.g., chest MRI).
  • Potential for Hallucination: Like other large models, MediSyn can still generate clinically inaccurate images, especially with unusual prompts.
  • No Formal Privacy Guarantees: While images are visually distinct, formal privacy guarantees are not yet provided.
  • Dataset Imbalance: The training dataset is imbalanced, with some modalities overrepresented, which could bias the model.

Future work suggested includes:

  • Developing strategies to address data imbalance and inter-modality competition.
  • Exploring methods for formal privacy guarantees without compromising visual quality.
  • Extending the model to 3D medical image and video generation.

Summary updated

Abstract

Diffusion models have recently gained significant traction due to their ability to generate high-fidelity and diverse images and videos conditioned on text prompts. In medicine, this application promises to address the critical challenge of data scarcity, a consequence of barriers in data sharing, stringent patient privacy regulations, and disparities in patient population and demographics. By generating realistic and varying medical 2D and 3D images, these models offer a rich, privacy-respecting resource for algorithmic training and research. To this end, we introduce MediSyn, a pair of instruction-tuned text-guided latent diffusion models with the ability to generate high-fidelity and diverse medical 2D and 3D images across specialties and modalities. Through established metrics, we show significant improvement in broad medical image and video synthesis guided by text prompts.
BibTeX
@misc{cho2024medisyn,
  author = {Joseph Cho and Cyril Zakka and Rohan Shad and Ross Wightman and Akshay Chaudhari and William Hiesinger},
  title = {{MediSyn: Text-Guided Diffusion Models for Broad Medical 2D and 3D Image Synthesis}},
  year = {2024},
  eprint = {2405.09806},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.09806},
}
Figure