A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations

Virginia Fernandez, Walter Hugo Lopez Pinaya, Pedro Borges, Mark S. Graham, Tom Vercauteren, M. Jorge Cardoso

TL;DR

This paper introduces brainSPADE3D, a novel 3D generative model capable of creating realistic multi-modal brain MRI images and their corresponding segmentation masks. This advancement is crucial for overcoming data scarcity in medical imaging, enabling the generation of diverse and controllable synthetic datasets that improve the robustness of downstream segmentation tasks, especially when encountering unexpected pathologies.

What Problem Does This Paper Address?

Deep learning models for medical image segmentation, while powerful, are heavily reliant on large, diverse, and well-annotated datasets. Acquiring such datasets for 3D medical imaging, particularly MRI, is challenging due to privacy regulations, the cost of expert annotation, and the inherent variability in pathologies and imaging protocols. Existing generative models often struggle to produce accurate segmentation masks alongside images, lack control over specific pathological features, and cannot easily generate images across different MRI contrasts (modalities). This limits their utility for data augmentation and for training models that can generalize to unseen or rare conditions.

Methods

The authors propose brainSPADE3D, a two-stage generative model.

  • Label Generator: This component utilizes a Latent Diffusion Model (LDM). It consists of a variational autoencoder (VAE) to compress images into a latent space and a diffusion model that operates within this latent space to generate segmentation masks. Crucially, this model incorporates a cross-attention mechanism to allow conditioning on specific disease types, enabling the generation of pathological phenotypes.
  • Image Generator: This part is a 3D adaptation of the SPADE (SPatially-Adaptive Denormalization) network. It takes the generated segmentation masks (labels) and conditions the image generation process. The encoder is kept 2D for parsimony, while the rest of the network is 3D to capture volumetric information. This generator is trained to produce multi-modal MRI images (T1, FLAIR, T2) corresponding to the input segmentation masks.

The model is designed to be conditional, allowing users to specify desired pathological phenotypes and contrasts for the generated synthetic data.

Datasets

The following datasets were used for training and evaluation:

  • SABREv3: A dataset of T1, FLAIR, and T2 MRI images.
  • ADNI2: A subset of T1 and FLAIR volumes.
  • BRATS: A dataset containing T1, FLAIR, and T2 volumes, often used for brain tumor segmentation.

The paper does not introduce a new benchmark dataset but rather focuses on generating synthetic data to augment existing ones.

Key Insights and Ideas

  • Joint Image and Segmentation Generation: The core innovation is the simultaneous generation of both high-fidelity MRI images and their corresponding accurate segmentation masks, addressing a major limitation of previous generative models for medical imaging.
  • Controllable Pathological Phenotypes: The ability to condition the generation on specific pathologies (e.g., White Matter Hyperintensities (WMH) and tumors) is a significant step towards creating targeted synthetic datasets for specific clinical tasks.
  • Improved Robustness to Unseen Pathologies: The generated synthetic data, particularly when combining pathologies, demonstrably improves the performance of segmentation models when faced with unexpected conditions (like tumors appearing in a dataset primarily focused on WMH).
  • 2mm vs. 1mm Resolution Trade-offs: The 2mm isotropic model showed better generalization to unseen combined pathologies compared to the 1mm model, suggesting that higher resolution might sometimes lead to a loss of semantic detail or increased difficulty in capturing complex interactions.

Key Results

  • High-Fidelity Generation: The brainSPADE3D model successfully generates realistic multi-modal MRI images and accurate segmentation masks, capable of combining different pathologies.
  • Improved Segmentation Performance: When trained on synthetic data generated by brainSPADE3D, segmentation models showed improved robustness and performance, particularly in scenarios with unexpected co-occurring pathologies (e.g., WMH segmentation in the presence of tumors). For instance, models trained with a mix of real and synthetic data achieved competitive WMH Dice scores and significantly reduced false positives (FPtum) on a challenging test set.
  • Label Quality: The generated labels showed comparable mean volumes for brain regions to real data, though some loss of detail was observed in thin structures like the subarachnoid CSF at 1mm resolution.

Limitations and Future Work

  • Resolution and Detail Loss: The 1mm³ model exhibited a loss of detail in thin structures like the subarachnoid CSF, and the model's ability to extrapolate to unseen phenotypes was lower compared to the 2mm³ model.
  • Harmonizing Semantic Domains: A gap exists between the synthetic and real image/label domains, which can affect downstream segmentation tasks. Future work could focus on harmonizing these domains.
  • Conditioning on More Variables: The authors suggest that conditioning on additional variables like age or ventricle size could introduce more variability and overcome limitations in tissue variability.
  • Scaling to More Pathologies: The method can be extended to more pathologies and tasks by fine-tuning on new pathological labels, leading to more generalizable segmentation models.
  • Diffusion Model Advancements: Future work could leverage recent advances in diffusion models for higher resolution and performance, potentially leading to even better synthetic labels with less domain shift.

Summary updated

Abstract

Generative modelling and synthetic data can be a surrogate for real medical imaging datasets, whose scarcity and difficulty to share can be a nuisance when delivering accurate deep learning models for healthcare applications. In recent years, there has been an increased interest in using these models for data augmentation and synthetic data sharing, using architectures such as generative adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the application of synthetic data to tasks such as 3D magnetic resonance imaging (MRI) segmentation remains limited due to the lack of labels associated with the generated images. Moreover, many of the proposed generative MRI models lack the ability to generate arbitrary modalities due to the absence of explicit contrast conditioning. These limitations prevent the user from adjusting the contrast and content of the images and obtaining more generalisable data for training task-specific models. In this work, we propose brainSPADE3D, a 3D generative model for brain MRI and associated segmentations, where the user can condition on specific pathological phenotypes and contrasts. The proposed joint imaging-segmentation generative model is shown to generate high-fidelity synthetic images and associated segmentations, with the ability to combine pathologies. We demonstrate how the model can alleviate issues with segmentation model performance when unexpected pathologies are present in the data.
BibTeX
@inproceedings{fernandez2023d,
  author = {Virginia Fernandez and Walter Hugo Lopez Pinaya and Pedro Borges and Mark S. Graham and Tom Vercauteren and M. Jorge Cardoso},
  title = {{A 3D Generative Model of Pathological Multi-modal MR Images and Segmentations}},
  booktitle = {DGM4MICCAI},
  year = {2023},
  url = {https://openreview.net/forum?id=hLNMQ59ckG},
}
Figure