Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network

Zizhao Zhang, Lin Yang, Yefeng Zheng

Abstract

This paper introduces a 3D Generative Adversarial Network (GAN) framework that simultaneously performs medical image translation (e.g., CT to MRI) and anatomical segmentation. A key contribution is the introduction of a shape-consistency loss, which ensures that critical anatomical structures are preserved during translation—a frequent failure point for standard unpaired image-to-image translation models like CycleGAN. By integrating generators and segmentors in a mutual bootstrapping process, the proposed method achieves significant improvements in both synthetic image quality and segmentation accuracy, particularly in data-scarce scenarios.

Introduction: Addressing Geometric Distortion

In medical imaging, obtaining paired datasets (the same patient scanned with both CT and MRI in identical positions) is exceptionally difficult. While models like CycleGAN can learn to translate between modalities using unpaired data, they often suffer from geometric distortion. Because standard cycle-consistency only requires that an image looks realistic and can be mapped back to the original, models frequently "hallucinate" changes in organ shape or position to satisfy the discriminator. In clinical practice, such distortions render synthetic images useless. Furthermore, training robust segmentation models typically requires massive labeled datasets, which are often unavailable for specific modalities.

Methods: Coupling Synthesis and Segmentation

The authors propose an end-to-end 3D framework that couples two primary tasks: Image Synthesis (via Generators) and Anatomical Segmentation (via Segmentors).

  1. 3D Cycle-Consistency: The framework extends standard CycleGAN to 3D to process holistic volumes. To mitigate "checkerboard artifacts," transpose convolutions are replaced with nearest-neighbor upsampling followed by standard convolutions.
  2. Shape-Consistency Loss ($L_{shape}$): This novel loss function uses segmentors ($S_A, S_B$) to evaluate synthetic images. If a generator distorts the anatomy during translation, the segmentor (trained on real shapes) produces a high error, forcing the generator to maintain spatial consistency.
  3. Mutual Bootstrapping: Generators and segmentors assist each other. Generators provide synthetic training data to segmentors in an online fashion, while segmentors provide the shape-consistency signal back to the generators.
The proposed training pipeline showing the interaction between generators, segmentors, and various loss functions, including the novel shape-consistency loss.

Key Insights

  • Geometric Ambiguity: The authors demonstrate that standard cycle-consistency is invariant to geometric transformations. The shape-consistency loss is essential to break this ambiguity and ensure anatomical fidelity.
  • Online vs. Offline Augmentation: Unlike traditional "ad-hoc" methods that use pre-generated synthetic images (offline), this paper advocates for online training. This co-evolution of generators and segmentors prevents the segmentor from being misled by poor-quality early-stage synthetic data.
  • The Importance of 3D: Processing entire volumes rather than 2D slices is crucial for maintaining spatial continuity across the Z-axis in medical imaging.

Key Results

The proposed method demonstrated significant improvements across multiple metrics:

  • Visual Quality: The model significantly reduced artifacts and preserved anatomical features (like the spine or aorta) better than standard 3D CycleGAN.
  • Shape Preservation (S-Score): The method achieved a higher segmentation score on synthetic images (69.2% for CT) compared to models without shape-consistency (66.8%).
  • Segmentation Performance: Online synthetic data augmentation significantly boosted Dice scores. CT segmentation improved from 67.8% (baseline) to 74.4%, while MRI segmentation improved from 70.3% to 73.2%.
  • Efficiency: The approach reached target accuracy using only 23% of the synthetic data required by traditional offline methods.
Visual comparison of CT-to-MRI and MRI-to-CT translation. The proposed method (Ours) shows superior anatomical preservation and fewer artifacts compared to standard CycleGAN.

Conclusion and Future Work

While the proposed framework successfully reduces geometric distortion and improves segmentation, some limitations remain. Training 3D CNNs is highly memory-intensive, which constrained the size of the segmentors used in this study. Additionally, while the gap between real and synthetic data was narrowed, further research into "gap-reduction" techniques is needed. Finally, applying the shape-consistency logic to more complex pathologies, such as tumors, would be a valuable extension of this work.

Summary updated

BibTeX
@inproceedings{zhang2018translating,
  author = {Zizhao Zhang and Lin Yang and Yefeng Zheng},
  title = {{Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2018},
  url = {https://openreview.net/forum?id=B1bMdTZ_-B},
}
Figure