SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth

Yuankai Huo, Zhoubing Xu, Hyeonsoo Moon, Shunxing Bao, Albert Assad, Tamara K. Moyo, Michael R. Savona, Richard G. Abramson, Bennett A. Landman

TL;DR

SynSeg-Net is an end-to-end framework that trains medical image segmentation models for a "target" modality (e.g., CT) without requiring any manual labels in that modality. By integrating unpaired image-to-image translation (CycleGAN) with a segmentation network, it allows researchers to leverage existing labels from a "source" modality (e.g., MRI) to achieve high-performance segmentation in another.

What Problem Does This Paper Address?

Deep convolutional neural networks (DCNNs) have revolutionized medical image segmentation, but they suffer from a major "generalizability" problem. A model trained to segment a spleen on MRI scans will typically fail when applied to CT scans due to differences in intensity distributions and contrast.

Traditionally, solving this requires clinicians to manually trace organs on hundreds of new scans for every new modality or disease cohort—a process that is incredibly labor-intensive and expensive. While "image synthesis" (turning an MRI into a synthetic CT) has been used as a workaround, previous approaches were often "two-stage": they synthesized the image first and then segmented it. This separation means the synthesis process isn't optimized for the final goal of accurate segmentation.

Methods

The authors propose SynSeg-Net, which combines cross-modality image synthesis and segmentation into a single, end-to-end trainable pipeline.

The Architecture

The network consists of two major components:

  1. Cycle Synthesis Subnet: Based on CycleGAN, this uses two generators ($G_1, G_2$) and two discriminators ($D_1, D_2$) to learn a mapping between the source and target modalities using unpaired images.
  2. Segmentation Subnet: A DCNN (specifically a 9-block ResNet) is concatenated directly after the source-to-target generator.
The SynSeg-Net architecture integrates cycle adversarial synthesis and segmentation into one framework.

Training Strategy

The model is trained using five loss functions simultaneously:

  • Adversarial Loss: Ensures synthesized images look realistic.
  • Cycle-Consistency Loss: Ensures that if you translate an image to the target modality and back, you get the original image.
  • Segmentation Loss: Calculated using the manual labels available only in the source modality, applied to the synthesized target images.

This joint optimization allows the synthesis engine to focus on preserving the structural features most important for segmentation.

Datasets

The authors evaluated SynSeg-Net on two distinct tasks:

  1. MRI-to-CT Splenomegaly (Abdominal):
    • Training: 60 MRI T2w scans and 19 CT scans from patients with enlarged spleens (splenomegaly).
  2. External Validation: 66 independent CT scans.
  3. CT-to-MRI TICV (Brain):
    • Data: 20 subjects with both MRI and CT scans. The task was to segment the Total Intracranial Volume (TICV).

Key Insights and Ideas

  • End-to-End is Superior: The primary "aha moment" is that training synthesis and segmentation together outperforms doing them sequentially. The joint training likely creates intermediate synthetic representations that are more "segmentation-friendly" than those produced by a standalone CycleGAN.
  • Cycle Consistency is Non-Negotiable: The authors tested a "half-cycle" version (SynSeg-Net-HC) without the target-to-source path. It performed significantly worse, proving that the bidirectional constraint is essential for maintaining structural integrity during translation.
  • Reusing Labels Across Modalities: The results suggest we can stop viewing different modalities as isolated silos. We can effectively "transfer" the human expertise embedded in MRI labels to the CT domain.

Key Results

SynSeg-Net achieved performance comparable to traditional supervised methods that did have access to target labels.

  • Splenomegaly Segmentation: SynSeg-Net reached a median Dice Similarity Coefficient (DSC) of 0.919, outperforming the two-stage CycleGAN+Seg method (0.880) and even slightly edging out a supervised ResNet trained directly on CT labels (0.911).
  • Brain TICV Segmentation: SynSeg-Net (DSC 0.966) again beat the two-stage approach (0.952), though it remained slightly behind the fully supervised baseline (0.982).
Qualitative comparison showing SynSeg-Net's ability to segment multiple organs without CT training labels.

Limitations and Future Work

  • 2D vs. 3D: The current implementation is 2D due to GPU memory constraints. The authors hypothesize that a 3D framework would yield even better spatial consistency if more training data were available.
  • Sampling Bias: Slices were sampled randomly; scans with more slices contribute more to training, which could introduce bias.
  • GAN Hallucinations: While effective for segmentation, the authors caution that synthesized images should not be used for direct clinical interpretation, as GANs can occasionally "hallucinate" or misrepresent fine-grained pathological details.

Future work includes exploring different network architectures (like U-Net) within the framework and evaluating inter-rater reliability by comparing the model against multiple human experts.

Summary updated

Abstract

A key limitation of deep convolutional neural network (DCNN)-based image segmentation methods is the lack of generalizability. Manually traced training images are typically required when segmenting organs in a new imaging modality or from distinct disease cohort. The manual efforts can be alleviated if the manually traced images in one imaging modality (e.g., MRI) are able to train a segmentation network for another imaging modality (e.g., CT). In this paper, we propose an end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentation network for a target imaging modality without having manual labels. SynSeg-Net is trained by using: 1) unpaired intensity images from source and target modalities and 2) manual labels only from source modality. SynSeg-Net is enabled by the recent advances of cycle generative adversarial networks and DCNN. We evaluate the performance of the SynSeg-Net on two experiments: 1) MRI to CT splenomegaly synthetic segmentation for abdominal images and 2) CT to MRI total intracranial volume synthetic segmentation for brain images. The proposed end-to-end approach achieved superior performance to two-stage methods. Moreover, the SynSeg-Net achieved comparable performance to the traditional segmentation network using target modality labels in certain scenarios. The source code of SynSeg-Net is publicly available.
BibTeX
@article{huo2019synseg,
  author = {Yuankai Huo and Zhoubing Xu and Hyeonsoo Moon and Shunxing Bao and Albert Assad and Tamara K. Moyo and Michael R. Savona and Richard G. Abramson and Bennett A. Landman},
  title = {{SynSeg-Net: Synthetic Segmentation Without Target Modality Ground Truth}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2019},
  url = {https://openreview.net/forum?id=qmtTcaBaZN},
}

Loading related content...

Figure