Multi-sequence generative adversarial network: better generation for enhanced magnetic resonance imaging images

Leizi Li, Jingchun Yu, Yijin Li, Jinbo Wei, Ruifang Fan, Dieen Wu, Yufeng YeFrontiers Comput. Neurosci 2024 · DOI

TL;DR

This paper introduces a novel multi-sequence generative adversarial network (GAN) that leverages multiple MRI sequences (T1W, T2W, FLAIR) to generate high-quality contrast-enhanced (T1CE) MRI images. This approach significantly reduces the need for contrast agents, benefiting patients who cannot receive them and potentially lowering healthcare costs.

What Problem Does This Paper Address?

Magnetic Resonance Imaging (MRI) is crucial for diagnosing conditions like brain diseases. However, a key diagnostic sequence, T1-weighted contrast-enhanced (T1CE) MRI, requires the injection of contrast agents. While effective, these agents carry risks, including adverse reactions and potential long-term side effects like nephrogenic systemic fibrosis (NSF), particularly in patients with renal insufficiency. Furthermore, certain patient groups, such as pregnant women and children, are contraindicated for contrast agents. This necessitates alternative methods to obtain T1CE images. While generative models like GANs show promise in image synthesis, existing methods often struggle with generating high-fidelity enhanced MRI images, especially when relying on single input sequences.

Methods

The authors propose a multi-sequence generative adversarial network (MS-GAN) built upon the pix2pix architecture. The core innovation lies in the generator's ability to accept multiple MRI sequences (e.g., T1W and T2W) as input. This multi-modal input is processed through specialized encoder modules (M1 and M2) that extract features at different levels. Crucially, the decoder then fuses these multi-dimensional features hierarchically, leveraging the complementarity of low-dimensional (detailed) and high-dimensional (semantic) information. This fusion strategy aims to create a more comprehensive representation for generating the target T1CE images. The discriminator remains a Markov discriminator (patch GAN) for efficiency. The model is trained using an adversarial loss and a pixel-wise L1 loss.

Datasets

The primary dataset used for training and evaluation is BraTS2021. This dataset contains MRI images from four modalities: T1W, T2W, FLAIR, and T1CE, along with brain tumor segmentation labels. The authors performed synthesis tasks generating T1CE images from various combinations of T1W, T2W, and FLAIR inputs.

Key Insights and Ideas

  • Multi-modal input is key: The paper strongly demonstrates that combining information from multiple non-enhanced MRI sequences (T1W, T2W, FLAIR) leads to significantly better generation of T1CE images compared to using single sequences.
  • Hierarchical feature fusion: The proposed generator's architecture, which gradually fuses multi-dimensional features from different modalities in the decoder, is critical for capturing richer information and improving generation quality.
  • T1W and T2W combination is optimal: The study identifies the combination of T1W and T2W images as the most effective input for generating T1CE images, outperforming other combinations and single-modal inputs.
  • Clinical relevance: The ability to generate T1CE images without contrast agents has significant implications for patient safety and accessibility of advanced MRI diagnostics.

Key Results

The proposed MS-GAN model consistently outperformed the baseline pix2pix model across all evaluation metrics (NMSE, RMSE, PSNR, SSIM). Specifically:

  • Quantitative improvements: The multi-modal input approach achieved higher PSNR and SSIM values and lower NMSE and RMSE values compared to single-modal inputs. For instance, the T1&T2 input combination yielded an average PSNR of 23.604, significantly higher than the pix2pix model's best single-modal input (T2 at 22.887).
  • Optimal combination: The T1&T2 input combination showed the best performance, with an average SSIM of 0.859 and an average PSNR of 23.604.
  • Clinical assessment: Radiologists rated the images generated by the T1&T2 multi-modal input model higher (average score of 3.36) than those from single-modal inputs (best average score of 2.94 for T2) and other multi-modal combinations.

Limitations and Future Work

  • GPU limitations: The authors acknowledge that GPU memory constraints limited the use of the full BraTS2021 dataset and the exploration of 3D networks, which could capture more spatial information.
  • Improvement margin: While multi-modal inputs improved generation, the improvement over single-modal inputs was noted as "small" in some aspects.
  • Uneven generation quality: GANs inherently can produce uneven quality, and the authors note that their model doesn't always reliably generate enhanced phase images.
  • Dataset preprocessing: The specific preprocessing of the BraTS2021 dataset might limit the model's applicability in clinical settings.

Future work suggestions include:

  • Utilizing multiple GPUs to enable 3D network architectures for better spatial feature capture.
  • Exploring diffusion models (e.g., stable diffusion) for enhanced generation capabilities.
  • Developing models for a wider range of tumors beyond gliomas.
  • Establishing a dedicated clinical dataset for more targeted model development and validation.

Summary updated

BibTeX
@article{li2024multi,
  author = {Leizi Li and Jingchun Yu and Yijin Li and Jinbo Wei and Ruifang Fan and Dieen Wu and Yufeng Ye},
  title = {{Multi-sequence generative adversarial network: better generation for enhanced magnetic resonance imaging images}},
  journal = {Frontiers Comput. Neurosci.},
  year = {2024},
  doi = {10.3389/fncom.2024.1365238},
}
Figure