MRI Image Generation Based on Text Prompts

Xinxian Fan, Mengye LyuarXiv 2025 · DOI · arXiv

TL;DR

Researchers have successfully fine-tuned Stable Diffusion to generate high-fidelity, controllable MRI images using simple text prompts. By leveraging datasets across different magnetic field strengths (0.3T and 3T), the team demonstrated that these synthetic images can significantly boost the performance of medical AI—increasing MRI modality classification accuracy from 71% to over 96% when real data is scarce.

What Problem Does This Paper Address?

Artificial Intelligence in medical imaging faces a massive "data bottleneck." High-quality MRI datasets are notoriously difficult to acquire due to exorbitant scanning costs, the rarity of specific pathologies, strict patient privacy regulations (HIPAA/GDPR), and the need for expensive expert annotations.

While generative models like GANs and Diffusion Models have previously been used to create "unconditional" MRI images (meaning the model just spits out a random brain scan), these lack semantic control. A researcher can't easily ask a standard model for a "3T T2-weighted axial slice at the mid-ventricle." This paper tackles the challenge of creating a controllable, text-guided pipeline that generates medically accurate images tailored to specific modalities and field strengths.

Methods

The authors built their framework on Stable Diffusion (v1.5), a latent diffusion model. They explored two primary fine-tuning strategies to bridge the gap between natural images (which SD was originally trained on) and specialized medical scans:

  1. DreamBooth: A technique typically used to "subject-bind" specific objects into a model. Here, it was used to bind medical concepts (like T1 or FLAIR) to rare token identifiers.
  2. UNet Fine-Tuning: The researchers specifically targeted the cross-attention layers and residual blocks within the SD UNet. This allowed the model to better map textual descriptions (field strength, modality, slice number) to the visual features of an MRI.

They employed an AdamW optimizer and experimented with different learning rates ($1 \times 10^{-4}$ vs. $5 \times 10^{-5}$) and training durations (1,000 to 10,000 steps) to find the "sweet spot" between image realism and output diversity.

Datasets

The study utilized three distinct data sources:

  • M4Raw: A low-field (0.3T) multi-contrast dataset containing T1, T2, and FLAIR images from 183 volunteers.
  • fastMRI: A widely used high-field (3T/1.5T) dataset. The authors selected 3T axial slices to represent the high-end of clinical imaging.
  • Huayin City Hospital Dataset: A small, "real-world" dataset of 0.35T scans used specifically for the downstream classification experiment to verify the utility of the synthetic data.

Preprocessing involved converting HDF5/DICOM formats to standardized 256x256 PNGs and filtering out unusable "empty" slices from the top and bottom of the scans.

Key Insights and Ideas

  • The "Physics" Gap: The base Stable Diffusion model understands what a "brain" looks like but lacks an understanding of MRI physics. Without fine-tuning, it produces brightly colored, structurally unrealistic images. Fine-tuning forces the model to respect the grayscale intensity distributions inherent to T1 (fat is bright) vs. T2 (fluid is bright).
  • Quality vs. Diversity Trade-off: The researchers found that as the Fréchet Inception Distance (FID) improved (indicating more realistic images), the Multi-Scale Structural Similarity Index (MS-SSIM) also increased. In generative AI, a higher MS-SSIM often indicates a decrease in diversity—the model starts "playing it safe" by generating very similar-looking slices.
  • Hyperparameter Sensitivity: For short training runs (1,000 steps), a higher learning rate ($1 \times 10^{-4}$) performed best. However, for longer runs (10,000 steps), a lower rate ($5 \times 10^{-5}$) was necessary to prevent overfitting and maintain a balance between quality and variety.

Key Results

  • Quantitative Improvement: Compared to the original SD model, the DreamBooth fine-tuned version saw the CLIP-FID drop from 39.94 to 17.63, a massive leap in semantic consistency and realism.
  • Visual Fidelity: The fine-tuned models successfully captured the nuances of different field strengths—retaining the characteristic "noise" of 0.3T scans while producing the "crisp" edges and high contrast of 3T scans.
  • Downstream Utility: In a practical test, a ResNet50 classifier was trained to identify MRI modalities (T1 vs. T2 vs. FLAIR). When training on a very small set of real 0.35T data, accuracy was only 71.15%. By augmenting the training set with synthetic images generated by their model, accuracy skyrocketed to 96.15%.

Limitations and Future Work

  • Healthy Brain Bias: The current study focused primarily on healthy brain anatomy. The authors acknowledge that for clinical utility, the model needs to be trained on pathological data (tumors, lesions, etc.).
  • Textual Nuance: The standard CLIP text encoder used in SD is trained on general internet text. The authors suggest that replacing this with a domain-specific encoder like RadBERT could allow for even finer control using complex radiological reports.
  • Information Loss: The use of central cropping during preprocessing led to some loss of peripheral anatomical information, which future iterations aim to fix using smarter scaling techniques.

Summary updated

Abstract

This study explores the use of text-prompted MRI image generation with the Stable Diffusion (SD) model to address challenges in acquiring real MRI datasets, such as high costs, limited rare case samples, and privacy concerns. The SD model, pre-trained on natural images, was fine-tuned using the 3T fastMRI dataset and the 0.3T M4Raw dataset, with the goal of generating brain T1, T2, and FLAIR images across different magnetic field strengths. The performance of the fine-tuned model was evaluated using quantitative metrics,including Fréchet Inception Distance (FID) and Multi-Scale Structural Similarity (MS-SSIM), showing improvements in image quality and semantic consistency with the text prompts. To further evaluate the model's potential, a simple classification task was carried out using a small 0.35T MRI dataset, demonstrating that the synthetic images generated by the fine-tuned SD model can effectively augment training datasets and improve the performance of MRI constrast classification tasks. Overall, our findings suggest that text-prompted MRI image generation is feasible and can serve as a useful tool for medical AI applications.
BibTeX
@misc{fan2025mri,
  author = {Xinxian Fan and Mengye Lyu},
  title = {{MRI Image Generation Based on Text Prompts}},
  year = {2025},
  eprint = {2505.22682},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.22682},
  howpublished = {arXiv},
}
Figure