Shape-Consistent Generative Adversarial Networks for Multi-Modal Medical Segmentation Maps

Leo Segre, Or Hirschorn, Dvir Ginzburg, Dan Raviv

TL;DR

Researchers from Tel Aviv University have developed a 3D shape-consistent Generative Adversarial Network (GAN) that significantly boosts medical image segmentation performance when training data is extremely scarce. By forcing the GAN to maintain anatomical integrity during image-to-image translation and utilizing 3D spatial augmentations, they achieved a massive 15.9% average accuracy increase in cardiac segmentation using only 16 training volumes.

What Problem Does This Paper Address?

Deep learning models are "data hungry," but in the medical world, high-quality labeled datasets are rare and expensive. This is especially true for multi-modal tasks (like segmenting the heart in both CT and MRI scans), where the same anatomy looks vastly different across modalities.

Existing solutions often use GANs to "translate" images from one modality to another to create synthetic training data. However, these methods frequently suffer from two flaws:

  1. 2D Limitations: Many models process 2D slices, losing the "Z-axis" spatial consistency crucial for 3D organs.
  2. Geometric Distortion: Standard GANs (like CycleGAN) are great at changing textures but often "hallucinate" or distort the actual shapes of organs, which ruins the training data for a segmentation model.

Methods

The authors propose a three-phase training pipeline centered around a 3D Shape-Consistent GAN:

  1. Phase 1: Segmentor Pre-training: A 3D U-Net segmentor is trained on the limited original data to get a "rough idea" of the anatomy.
  2. Phase 2: Shape-Consistent Translation: They train a 3D CycleGAN to translate between CT and MRI. Crucially, they introduce a Spatial Loss. The pre-trained segmentor from Phase 1 acts as a judge: if the GAN translates a CT heart into an MRI heart but changes its shape so much that the segmentor can no longer recognize the parts, the GAN is penalized. This ensures the synthetic data remains anatomically accurate.
  3. Phase 3: Final Training: The segmentor is re-trained from scratch using a "boosted" dataset containing both the original scans and the high-quality synthetic 3D volumes generated in Phase 2.

To further help the model generalize, they used the TorchIO library to apply heavy 3D spatial augmentations, such as random elastic deformations and anisotropy simulations, creating a total of 400 augmented 3D scans from the original 16.

Datasets

The study utilized the Multi-Modality Whole Heart Segmentation (MMWHS) Challenge 2017 dataset.

  • Composition: 20 CT and 20 MRI unpaired volumes.
  • Labels: 7 cardiac structures (e.g., ascending aorta, left/right ventricles, atria).
  • Split: 80% for training (16 volumes) and 20% for testing.

Key Insights and Ideas

  • The Power of 3D: Moving from 2D slices to 3D volumes provides "implicit smoothness." The model learns the volumetric relationship between slices, leading to much cleaner segmentation maps.
  • Segmentors as Regularizers: Using a segmentation network to constrain a GAN is a clever way to enforce "semantic consistency." It prevents the GAN from simply "painting" MRI textures onto random shapes; it must respect the underlying biological structure.
  • Synergy of Augmentation: The paper shows that traditional spatial augmentations (stretching/warping) and GAN-based style augmentations (CT-to-MRI) are complementary. Together, they allow a model to learn robust features from a tiny sample size.

Key Results

The proposed method outperformed several State-of-the-Art (SOTA) unsupervised domain adaptation methods by a wide margin:

Model CT Dice Score (%) MRI Dice Score (%)
SIFA (Previous SOTA) 63.4 74.1
Ours (4 labels) 88.2 81.2
Ours (7 labels) 85.0 81.8

The jump from ~63% to ~88% in CT segmentation is particularly striking, demonstrating that 3D context and shape consistency are game-changers for small-scale medical AI.

Limitations and Future Work

  • MRI Diversity: The authors noted that MRI results were slightly less consistent than CT. This is likely because the MRI scans in the dataset had more varied orientations and sizes, making it harder for the GAN to produce perfectly homogeneous synthetic data.
  • Computational Cost: Training 3D GANs and 3D U-Nets is computationally expensive compared to 2D versions, requiring significant GPU memory (the authors used 8 NVIDIA Quadro RTX 8000s).
  • Future Direction: The authors suggest investigating better ways to generate synthetic MRI scans that account for the higher variance in acquisition protocols across different hospitals.

Summary updated

Abstract

Image translation across domains for unpaired datasets has gained interest and great improvement lately. In medical imaging, there are multiple imaging modalities, with very different characteristics. Our goal is to use cross-modality adaptation between CT and MRI whole cardiac scans for semantic segmentation. We present a segmentation network using synthesised cardiac volumes for extremely limited datasets. Our solution is based on a 3D cross-modality generative adversarial network to share information between modalities and generate synthesized data using unpaired datasets. Our network utilizes semantic segmentation to improve generator shape consistency, thus creating more realistic synthesised volumes to be used when re-training the segmentation network. We show that improved segmentation can be achieved on small datasets when using spatial augmentations to improve a generative adversarial network. These augmentations improve the generator capabilities, thus enhancing the performance of the Segmentor. Using only 16 CT and 16 MRI cardiovascular volumes, improved results are shown over other segmentation methods while using the suggested architecture. Our code is publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
BibTeX
@inproceedings{segre2022shape,
  author = {Leo Segre and Or Hirschorn and Dvir Ginzburg and Dan Raviv},
  title = {{Shape-Consistent Generative Adversarial Networks for Multi-Modal Medical Segmentation Maps}},
  booktitle = {ISBI},
  year = {2022},
  url = {https://openreview.net/forum?id=j9WoVhaWUp},
}

Loading related content...

Figure