DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis

Hongwei Li, Johannes C. Paetzold, Anjany Sekuboyina, Florian Kofler, Jianguo Zhang, Jan S. Kirschke, Benedikt Wiestler, Bjoern H. MenzeMICCAI 2019 · DOI · OpenReview

TL;DR

This paper introduces DiamondGAN, a novel generative adversarial network capable of synthesizing missing or low-quality MRI sequences from other available modalities, even when they are not perfectly aligned. This unified approach significantly improves the clinical utility of MRI data by enabling more comprehensive analysis and reducing the need for repeated scans.

What Problem Does This Paper Address?

Magnetic Resonance Imaging (MRI) is a powerful diagnostic tool, but clinical practice often involves acquiring multiple MRI sequences (modalities) to capture different tissue properties. However, these datasets can be incomplete due to various reasons like patient movement, scanner limitations, or differing acquisition protocols across institutions. This incompleteness hinders comprehensive analysis, especially for longitudinal studies crucial for tracking diseases like multiple sclerosis. Existing methods for synthesizing missing MRI sequences often struggle with:

  • Non-aligned Modalities: MRI sequences are rarely perfectly spatially aligned, making traditional regression-based methods difficult to apply.
  • Missing Modalities: If certain modalities are missing, existing methods might be restricted to using only the smallest common subset of available data or require complex iterative infilling.
  • Scalability: Training separate models for every possible combination of input modalities is computationally expensive and impractical.

This paper aims to address these challenges by proposing a unified, scalable, and flexible approach to multi-modal MRI synthesis.

Methods

The core of the proposed solution is DiamondGAN, a unified, multi-modal generative adversarial network (GAN). Its key innovations include:

  • Unified Multi-Modal Synthesis: DiamondGAN learns a "diamond-shape" topology, enabling it to synthesize a target modality from any subset of available input modalities within a single model. This is achieved using a pair of generators (G1 and G2) and discriminators (D1 and D2).
  • Availability Condition: A binary condition vector c is introduced, indicating which input modalities are present. This condition is spatially replicated and concatenated with the input images, allowing the model to flexibly handle missing modalities during training and inference.
  • Multi-Modal Cycle-Consistency Loss: To address the non-alignment issue, the authors extend the traditional cycle-consistency loss. This loss ensures that the synthesized target modality preserves the content of its input modalities, even when they are not spatially aligned.
  • Two-Stream Network Architecture: The network utilizes an encoder-decoder structure with two streams that process input modalities and their availability conditions separately before merging them at the feature level.
  • Training Stabilisation: Techniques like least-squares GAN loss and a history of generated images for discriminator updates are employed to stabilize training.

Datasets

The paper utilizes two datasets:

  • Dataset 1: A local hospital dataset comprising 65 scans of patients with Multiple Sclerosis (MS) lesions. This dataset includes co-registered Flair, T1, T2, Double Inversion Recovery (DIR), and Contrast-Enhanced T1 (T1-c) sequences. This dataset is used for training and evaluating the synthesis of DIR and T1-c from common modalities (Flair, T1, T2).
  • Dataset 2: A subset of the public MICCAI-WMH dataset, containing 40 subjects with Flair and T1 modalities. This dataset is used to demonstrate the approach's applicability to multiple datasets with incomplete and non-aligned modalities.

All slices are cropped/padded to 240x240 and intensity values are rescaled to [-1, 1].

Key Insights and Ideas

  • Synergy of Modalities: The research highlights that combining information from multiple MRI modalities, even if not perfectly aligned, leads to significantly better synthesis quality than using single modalities.
  • Flexibility is Key: The ability to handle arbitrary subsets of input modalities and the "missing modality" problem is crucial for real-world clinical applications where data availability varies.
  • Radiologist Validation: The most compelling insight is the validation by experienced neuroradiologists, who found synthetic DIR images to be indistinguishable from real ones, demonstrating the clinical relevance and high fidelity of DiamondGAN.
  • DIR Synthesis is Challenging but Achievable: Synthesizing DIR sequences, which are complex and contain proprietary information, benefits significantly from multi-modal inputs, and DiamondGAN achieves comparable plausibility to real DIR images.

Key Results

  • Quantitative Evaluation: DiamondGAN consistently outperforms a baseline CycleGAN model in terms of PSNR and MAE for both DIR and T1-c synthesis. For DIR synthesis, using a combined T1+T2+Flair input yielded the best results (MAE 0.058). For T1-c synthesis, inputs involving the T1 modality (MAE 0.045-0.048) performed best, aligning with the nature of T1-c scans.
  • Qualitative Evaluation: Visual inspection of synthetic images shows preservation of tissue contrast and anatomical information.
  • Radiologist Evaluation: Fourteen neuroradiologists rated synthetic images on a 6-star scale. For DIR synthesis with T1+T2+Flair input, the rating scores of synthetic images were not significantly different from real DIR images (p-value = 0.1432), indicating indistinguishability. For T1-c synthesis, scores were consistently good when T1 modality was included.

Limitations and Future Work

  • Alignment Error Impact: The paper notes that more differences were observed in synthetic DIR images compared to T1-c, potentially due to residual alignment errors from registration methods, especially around the brain boundary.
  • Future Work: While not explicitly detailed, the success in synthesizing clinically relevant sequences like DIR and T1-c suggests future work could focus on synthesizing other complex or rare MRI sequences, further improving robustness to alignment issues, and exploring applications in specific disease detection tasks (as hinted by the MS lesion detection study). The potential to reduce medical costs by avoiding repeated scans is also a significant avenue for future impact.

Summary updated

Abstract

Synthesizing MR imaging sequences is highly relevant in clinical practice, as single sequences are often missing or are of poor quality (e.g. due to motion). Naturally, the idea arises that a target modality would benefit from multi-modal input, as proprietary information of individual modalities can be synergistic. However, existing methods fail to scale up to multiple non-aligned imaging modalities, facing common drawbacks of complex imaging sequences. We propose a novel, scalable and multi-modal approach called DiamondGAN. Our model is capable of performing exible non-aligned cross-modality synthesis and data infill, when given multiple modalities or any of their arbitrary subsets, learning structured information in an end-to-end fashion. We synthesize two MRI sequences with clinical relevance (i.e., double inversion recovery (DIR) and contrast-enhanced T1 (T1-c)), reconstructed from three common sequences. In addition, we perform a multi-rater visual evaluation experiment and find that trained radiologists are unable to distinguish synthetic DIR images from real ones.
BibTeX
@inproceedings{li2019diamondgan,
  author = {Hongwei Li and Johannes C. Paetzold and Anjany Sekuboyina and Florian Kofler and Jianguo Zhang and Jan S. Kirschke and Benedikt Wiestler and Bjoern H. Menze},
  title = {{DiamondGAN: Unified Multi-Modal Generative Adversarial Networks for MRI Sequences Synthesis}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  doi = {10.1007/978-3-030-32251-9\_87},
}
Figure