Shape-consistent Generative Adversarial Networks for multi-modal Medical segmentation maps

Leo Segre, Or Hirschorn, Dvir Ginzburg, Dan Raviv

Abstract

Image translation across domains for unpaired datasets has gained interest and great improvement lately. In medical imaging, there are multiple imaging modalities, with very different characteristics. Our goal is to use cross-modality adaptation between CT and MRI whole cardiac scans for semantic segmentation. We present a segmentation network using synthesised cardiac volumes for extremely limited datasets. Our solution is based on a 3D cross-modality generative adversarial network to share information between modalities and generate synthesized data using unpaired datasets. Our network utilizes semantic segmentation to improve generator shape consistency, thus creating more realistic synthesised volumes to be used when re-training the segmentation network. We show that improved segmentation can be achieved on small datasets when using spatial augmentations to improve a generative adversarial network. These augmentations improve the generator capabilities, thus enhancing the performance of the Segmentor. Using only 16 CT and 16 MRI cardiovascular volumes, improved results are shown over other segmentation methods while using the suggested architecture.

Introduction

Deep learning models are notoriously "data hungry," yet high-quality labeled datasets in medical imaging are rare and expensive to produce. This challenge is amplified in multi-modal tasks, such as segmenting cardiac structures in both CT and MRI scans, where the same anatomy exhibits vastly different intensities and textures. While Generative Adversarial Networks (GANs) are often used to translate images between modalities to create synthetic training data, standard approaches frequently suffer from geometric distortions and a lack of 3D spatial consistency. This paper introduces a 3D shape-consistent GAN designed to maintain anatomical integrity during translation, specifically optimized for extremely limited datasets.

Methods

The proposed architecture utilizes a three-phase training pipeline to ensure that synthetic data remains anatomically accurate and useful for segmentation.

The three-phase training pipeline: Phase 1 pre-trains the segmentor; Phase 2 trains the shape-consistent GAN with spatial loss; Phase 3 performs final segmentor training.
  1. Phase 1: Segmentor Pre-training: A 3D U-Net segmentor ($S_A$) is trained on the limited original data to establish a baseline understanding of the anatomy.
  2. Phase 2: Shape-Consistent Translation: A 3D CycleGAN translates images between CT and MRI. To prevent the "hallucination" of structures, a Spatial Loss is introduced. The pre-trained segmentor from Phase 1 acts as a regularizer; if the GAN's translation distorts the organ's shape such that the segmentor can no longer recognize it, the GAN is penalized.
  3. Phase 3: Final Training: The segmentor is re-trained from scratch using a combined dataset of original scans and the high-quality synthetic 3D volumes generated in Phase 2.

To maximize the utility of the small dataset (16 volumes), the researchers employed heavy 3D spatial augmentations via the TorchIO library, including random elastic deformations and anisotropy simulations, expanding the training set to 400 augmented 3D scans.

Results and Discussion

The study utilized the Multi-Modality Whole Heart Segmentation (MMWHS) Challenge 2017 dataset, consisting of 20 CT and 20 MRI unpaired volumes. The proposed method demonstrated a significant performance leap over existing unsupervised domain adaptation methods.

Qualitative segmentation results for CT (top) and MRI (bottom) showing the progression from original scans to final segmentation maps.

As shown in the table below, the model achieved a massive 15.9% average accuracy increase in cardiac segmentation compared to previous state-of-the-art (SOTA) methods like SIFA.

Model CT Dice Score (%) MRI Dice Score (%)
SIFA (Previous SOTA) 63.4 74.1
Ours - 4 labels 88.2 81.2
Ours - 7 labels 85.0 81.8

The transition from 2D slice processing to 3D volumetric analysis provided implicit smoothness and better capture of the relationship between slices. The use of a segmentor as a regularizer successfully enforced semantic consistency, ensuring the GAN respected underlying biological structures rather than just applying textures to arbitrary shapes.

Conclusion

By combining 3D spatial context with shape-consistent generative modeling, this research demonstrates that high-accuracy medical segmentation is possible even with very small datasets. While MRI results showed slightly more variance due to acquisition protocol diversity, the overall framework provides a robust solution for cross-modality adaptation. Future work will focus on further improving synthetic MRI generation to account for higher variance in clinical settings.

Summary updated

Abstract

Image translation across domains for unpaired datasets has gained interest and great improvement lately. In medical imaging, there are multiple imaging modalities, with very different characteristics. Our goal is to use cross-modality adaptation between CT and MRI whole cardiac scans for semantic segmentation. We present a segmentation network using synthesised cardiac volumes for extremely limited datasets. Our solution is based on a 3D cross-modality generative adversarial network to share information between modalities and generate synthesized data using unpaired datasets. Our network utilizes semantic segmentation to improve generator shape consistency, thus creating more realistic synthesised volumes to be used when re-training the segmentation network. We show that improved segmentation can be achieved on small datasets when using spatial augmentations to improve a generative adversarial network. These augmentations improve the generator capabilities, thus enhancing the performance of the Segmentor. Using only 16 CT and 16 MRI cardiovascular volumes, improved results are shown over other segmentation methods while using the suggested architecture.
BibTeX
@inproceedings{segre2022shape,
  author = {Leo Segre and Or Hirschorn and Dvir Ginzburg and Dan Raviv},
  title = {{Shape-consistent Generative Adversarial Networks for multi-modal Medical segmentation maps}},
  booktitle = {ISBI},
  year = {2022},
  url = {https://openreview.net/forum?id=j9WoVhaWUp},
}
Figure