Abstract
Image translation across domains for unpaired datasets has gained interest and great improvement lately. In medical imaging, there are multiple imaging modalities, with very different characteristics. Our goal is to use cross-modality adaptation between CT and MRI whole cardiac scans for semantic segmentation. We present a segmentation network using synthesised cardiac volumes for extremely limited datasets. Our solution is based on a 3D cross-modality generative adversarial network to share information between modalities and generate synthesized data using unpaired datasets. Our network utilizes semantic segmentation to improve generator shape consistency, thus creating more realistic synthesised volumes to be used when re-training the segmentation network. We show that improved segmentation can be achieved on small datasets when using spatial augmentations to improve a generative adversarial network. These augmentations improve the generator capabilities, thus enhancing the performance of the Segmentor. Using only 16 CT and 16 MRI cardiovascular volumes, improved results are shown over other segmentation methods while using the suggested architecture.
Introduction
Deep learning models are notoriously "data hungry," yet high-quality labeled datasets in medical imaging are rare and expensive to produce. This challenge is amplified in multi-modal tasks, such as segmenting cardiac structures in both CT and MRI scans, where the same anatomy exhibits vastly different intensities and textures. While Generative Adversarial Networks (GANs) are often used to translate images between modalities to create synthetic training data, standard approaches frequently suffer from geometric distortions and a lack of 3D spatial consistency. This paper introduces a 3D shape-consistent GAN designed to maintain anatomical integrity during translation, specifically optimized for extremely limited datasets.
Methods
The proposed architecture utilizes a three-phase training pipeline to ensure that synthetic data remains anatomically accurate and useful for segmentation.
- Phase 1: Segmentor Pre-training: A 3D U-Net segmentor ($S_A$) is trained on the limited original data to establish a baseline understanding of the anatomy.
- Phase 2: Shape-Consistent Translation: A 3D CycleGAN translates images between CT and MRI. To prevent the "hallucination" of structures, a Spatial Loss is introduced. The pre-trained segmentor from Phase 1 acts as a regularizer; if the GAN's translation distorts the organ's shape such that the segmentor can no longer recognize it, the GAN is penalized.
- Phase 3: Final Training: The segmentor is re-trained from scratch using a combined dataset of original scans and the high-quality synthetic 3D volumes generated in Phase 2.
To maximize the utility of the small dataset (16 volumes), the researchers employed heavy 3D spatial augmentations via the TorchIO library, including random elastic deformations and anisotropy simulations, expanding the training set to 400 augmented 3D scans.
Results and Discussion
The study utilized the Multi-Modality Whole Heart Segmentation (MMWHS) Challenge 2017 dataset, consisting of 20 CT and 20 MRI unpaired volumes. The proposed method demonstrated a significant performance leap over existing unsupervised domain adaptation methods.
As shown in the table below, the model achieved a massive 15.9% average accuracy increase in cardiac segmentation compared to previous state-of-the-art (SOTA) methods like SIFA.
| Model | CT Dice Score (%) | MRI Dice Score (%) |
|---|---|---|
| SIFA (Previous SOTA) | 63.4 | 74.1 |
| Ours - 4 labels | 88.2 | 81.2 |
| Ours - 7 labels | 85.0 | 81.8 |
The transition from 2D slice processing to 3D volumetric analysis provided implicit smoothness and better capture of the relationship between slices. The use of a segmentor as a regularizer successfully enforced semantic consistency, ensuring the GAN respected underlying biological structures rather than just applying textures to arbitrary shapes.
Conclusion
By combining 3D spatial context with shape-consistent generative modeling, this research demonstrates that high-accuracy medical segmentation is possible even with very small datasets. While MRI results showed slightly more variance due to acquisition protocol diversity, the overall framework provides a robust solution for cross-modality adaptation. Future work will focus on further improving synthetic MRI generation to account for higher variance in clinical settings.
Summary updated