Abstract
A key limitation of deep convolutional neural networks (DCNN) based image segmentation methods is the lack of generalizability. Manually traced training images are typically required when segmenting organs in a new imaging modality or from distinct disease cohort. The manual efforts can be alleviated if the manually traced images in one imaging modality (e.g., MRI) are able to train a segmentation network for another imaging modality (e.g., CT). In this paper, we propose an end-to-end synthetic segmentation network (SynSeg-Net) to train a segmentation network for a target imaging modality without having manual labels. SynSeg-Net is trained by using (1) unpaired intensity images from source and target modalities, and (2) manual labels only from source modality. SynSeg-Net is enabled by the recent advances of cycle generative adversarial networks (CycleGAN) and DCNN. We evaluate the performance of the SynSeg-Net on two experiments: (1) MRI to CT splenomegaly synthetic segmentation for abdominal images, and (2) CT to MRI total intracranial volume synthetic segmentation (TICV) for brain images. The proposed end-to-end approach achieved superior performance to two stage methods. Moreover, the SynSeg-Net achieved comparable performance to the traditional segmentation network using target modality labels in certain scenarios.
Introduction
Deep convolutional neural networks (DCNNs) have set new benchmarks in medical image segmentation. However, their reliance on large, manually annotated datasets remains a significant bottleneck. A model trained on one modality, such as Magnetic Resonance Imaging (MRI), typically fails to generalize to another, such as Computed Tomography (CT), due to differences in intensity distributions and contrast. This "generalizability problem" forces researchers to undertake the labor-intensive task of manual tracing for every new imaging modality or disease cohort.
In this work, we present SynSeg-Net, an end-to-end synthetic segmentation network designed to train segmentation models for a target modality without requiring any manual labels in that domain. By leveraging unpaired images from both source and target modalities alongside existing labels from the source modality, SynSeg-Net effectively "transfers" segmentation capabilities across domains.
Methods
The core innovation of SynSeg-Net lies in its integration of cross-modality image synthesis and segmentation into a single, jointly optimized pipeline.
Architecture
The framework consists of two primary sub-networks:
- Cycle Synthesis Subnet: Built upon the CycleGAN architecture, this subnet employs two generators ($G_1, G_2$) and two discriminators ($D_1, D_2$). It learns a bidirectional mapping between the source and target modalities using unpaired intensity images.
- Segmentation Subnet: A DCNN (specifically a 9-block ResNet architecture) is appended to the source-to-target generator. This allows the network to perform segmentation directly on the synthesized target-modality images.
Training Strategy
Unlike traditional "two-stage" methods where synthesis and segmentation are trained independently, SynSeg-Net uses an end-to-end training strategy. The model is optimized using five concurrent loss functions: adversarial loss to ensure realism, cycle-consistency loss to preserve structural integrity, and segmentation loss calculated using the manual labels available in the source modality. This joint optimization ensures that the synthesis process is specifically tuned to preserve the anatomical structures necessary for accurate segmentation.
Experimental Evaluation
The performance of SynSeg-Net was validated across two distinct tasks:
- MRI-to-CT Splenomegaly Segmentation: Using 60 MRI T2w scans and 19 CT scans from patients with enlarged spleens, the model was trained to segment the spleen on CT scans. Validation was performed on 66 independent CT scans.
- CT-to-MRI Total Intracranial Volume (TICV): Using 20 subjects with both MRI and CT scans, the model learned to segment TICV in MRI scans using CT labels.
Results
SynSeg-Net demonstrated superior performance compared to two-stage methods and achieved results comparable to fully supervised models trained with target-modality labels.
In the splenomegaly task, SynSeg-Net achieved a median Dice Similarity Coefficient (DSC) of 0.919, significantly outperforming the two-stage CycleGAN+Seg approach (0.880). Notably, it slightly outperformed a supervised ResNet trained directly on CT labels (0.911). For the brain TICV task, SynSeg-Net reached a DSC of 0.966, again surpassing the two-stage baseline (0.952).
Discussion and Conclusion
The success of SynSeg-Net highlights the power of end-to-end training in domain adaptation. By coupling synthesis and segmentation, the network creates intermediate representations that are optimized for the final clinical task rather than just visual fidelity. The inclusion of cycle-consistency was found to be critical; "half-cycle" variants without the bidirectional constraint performed significantly worse, likely due to structural distortions during translation.
While the current 2D implementation is constrained by GPU memory, it provides a robust proof-of-concept for cross-modality label reuse. Future work will explore 3D implementations and the integration of different segmentation backbones to further enhance performance. SynSeg-Net represents a significant step toward reducing the manual annotation burden in medical imaging.
Summary updated