Lesion Mask-based Simultaneous Synthesis of Anatomic and MolecularMR Images using a GAN

Pengfei Guo, Puyang Wang, Jinyuan Zhou, Vishal M. Patel, Shanshan Jiang

TL;DR

This paper introduces SAMR, a novel Generative Adversarial Network (GAN) that can synthesize multiple types of MRI scans (anatomic and molecular) simultaneously from a lesion mask. This addresses the critical shortage of annotated medical imaging data for training AI models, particularly for challenging conditions like malignant gliomas.

What Problem Does This Paper Address?

Developing AI models for medical image analysis, especially for complex diseases like malignant gliomas, relies heavily on large, diverse, and well-annotated datasets. However, acquiring such datasets is a significant bottleneck due to the labor-intensive, time-consuming, and expensive nature of expert annotation. While traditional data augmentation techniques (flipping, scaling, etc.) offer some help, they fail to generate truly diverse image content. This paper tackles the problem of generating synthetic, yet realistic, multi-modal MRI scans that can effectively augment existing datasets, thereby improving the performance of downstream AI tasks like lesion segmentation.

Methods

The core of the proposed method is a conditional GAN named SAMR (Synthesis of Anatomic and Molecular MR Images network). SAMR's generator is designed to take a lesion segmentation mask as input and produce multiple MRI sequences (T1w, Gd-T1w, T2w, FLAIR, and APTw) simultaneously. Key components include:

  • Stretch-out Up-sampling Module: This module is responsible for generating the synthesized MRI slices from latent representations.
  • Brain Atlas Encoder: This component incorporates anatomical prior information from a brain atlas to guide the synthesis process.
  • Segmentation Consistency Module: A U-net based segmentation module is used to enforce consistency between the generated images and the input lesion masks, ensuring that the synthesized lesions maintain their shape and characteristics. This is achieved through a lesion shape consistency loss ($L_c$).
  • Multi-scale Label-wise Discriminators: Instead of a single discriminator, SAMR employs multiple discriminators that operate on different scales and focus on specific regions of interest (ROIs) within the image (background, normal brain, and lesion). This provides more targeted supervision and improves the realism of synthesized images within these critical areas.

The overall training objective is a multi-task loss function that combines adversarial loss ($L_{GAN}$), feature matching loss ($L_{FM}$), and the lesion shape consistency loss ($L_c$).

Datasets

The study utilized MRI scans from 90 postsurgical patients. Anatomic sequences (T1w, Gd-T1w, T2w, FLAIR) and molecular APTw sequences were acquired. Expert neuroradiologists manually annotated five labels (background, normal brain, edema, cavity, tumor) for each patient. A multivariate template construction tool was used to create a group average atlas. The dataset was preprocessed and split into 1080 instances for training and 270 for testing.

Key Insights and Ideas

  • Simultaneous Multi-modal Synthesis: The ability to generate multiple MRI sequences from a single lesion mask is a significant advancement, reducing the complexity and data requirements for generating comprehensive synthetic datasets.
  • Leveraging Brain Atlases for Prior: Incorporating a brain atlas provides crucial anatomical context, leading to more realistic and anatomically plausible synthesized images, especially when dealing with complex brain structures.
  • Targeted Supervision with Label-wise Discriminators: Focusing discriminators on specific ROIs (lesion, normal brain, background) allows for more precise control over image quality and realism in critical areas, outperforming holistic image discrimination.
  • Shape Consistency for Lesion Realism: The lesion shape consistency loss is vital for ensuring that synthesized lesions accurately reflect the input mask, preventing unrealistic distortions or blurring.

Key Results

  • Qualitative Comparison: The proposed SAMR method produced significantly more realistic and anatomically accurate MRI sequences compared to existing state-of-the-art methods like pix2pix, pix2pixHD, and Shin et al. The synthesized images exhibited fewer artifacts and more accurate radiographic features, particularly for the APTw sequence.
  • Human Perceptual Study: An expert neuroradiologist rated the images generated by SAMR as the most preferred, demonstrating the clinical practicality and realism of the synthesized data. SAMR achieved a preference rate of 72.1% compared to 65.6% for the real images and lower rates for other methods.
  • Data Augmentation Performance: When used to augment training data for lesion segmentation, SAMR significantly improved performance across various metrics (Dice score, Hausdorff distance, Sensitivity, Specificity) compared to baseline (real data only) and other synthesis methods. For instance, with 50% synthesized data, SAMR achieved a Dice score of 0.821 for tumors, outperforming Shin et al. (0.772) and others.
  • Robustness to Manipulations: The method demonstrated robustness to various lesion mask manipulations, including mirroring, size changes, and even reassembling lesion information from different patients, highlighting its flexibility.

Limitations and Future Work

  • Limitations: The paper acknowledges that the synthesized data can degrade segmentation performance when used in excess of real data (as seen in Table 2, Exp. 2, where 75% real data was used). The current method is limited to 2D synthesis, and extending it to 3D would be beneficial.
  • Future Work: The authors plan to generalize data augmentation by synthesis to other tasks, such as classification. They also aim to extend the proposed method to 3D synthesis, which would require more extensive 3D molecular MRI data for training.

Summary updated

Abstract

Data-driven automatic approaches have demonstrated their great potential in resolving various clinical diagnostic dilemmas for patients with malignant gliomas in neuro-oncology with the help of conventional and advanced molecular MR images. However, the lack of sufficient annotated MRI data has vastly impeded the development of such automatic methods. Conventional data augmentation approaches, including flipping, scaling, rotation, and distortion are not capable of generating data with diverse image content. In this paper, we propose a method, called synthesis of anatomic and molecular MR images network (SAMR), which can simultaneously synthesize data from arbitrary manipulated lesion information on multiple anatomic and molecular MRI sequences, including T1-weighted (T1w), gadolinium enhanced T1w (Gd-T1w), T2-weighted (T2w), fluid-attenuated inversion recovery (FLAIR), and amide proton transfer-weighted (APTw). The proposed framework consists of a stretch-out up-sampling module, a brain atlas encoder, a segmentation consistency module, and multi-scale label-wise discriminators. Extensive experiments on real clinical data demonstrate that the proposed model can perform significantly better than the state-of-the-art synthesis methods.
BibTeX
@inproceedings{guo2020lesion,
  author = {Pengfei Guo and Puyang Wang and Jinyuan Zhou and Vishal M. Patel and Shanshan Jiang},
  title = {{Lesion Mask-based Simultaneous Synthesis of Anatomic and MolecularMR Images using a GAN}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2020},
  url = {https://openreview.net/forum?id=TGHxiTijl-0o},
}
Figure