DRL-STNet: Unsupervised Domain Adaptation for Cross-modality Medical Image Segmentation via Disentangled Representation Learning

Hui Lin, Florian Schiffers, Santiago López-Tapia, Neda Tavakoli, Daniel Kim, Aggelos KatsaggelosSubmission to FLARE 2024 · Rejected

Abstract

Cross-modality medical image segmentation is often hindered by the "domain shift" between imaging modalities, such as CT and MRI. This paper introduces DRL-STNet, a novel framework designed for Unsupervised Domain Adaptation (UDA). By integrating Disentangled Representation Learning (DRL) for high-fidelity image translation with an iterative self-training (ST) loop, the framework enables the transfer of knowledge from labeled source domains to unlabeled target domains without requiring paired data. DRL-STNet achieves a significant performance leap, outperforming previous state-of-the-art methods by 11.4% in Dice Similarity Coefficient (DSC) for abdominal organ segmentation.

Introduction

In medical artificial intelligence, a model trained on one imaging modality (e.g., CT) frequently fails when applied to another (e.g., MRI) because the visual "style" differs significantly despite identical underlying anatomy. This phenomenon, known as domain shift, typically requires the collection and manual annotation of thousands of new images for the target modality—a process that is both labor-intensive and expensive. Furthermore, obtaining paired datasets where the same patient receives both scans simultaneously is logistically challenging. DRL-STNet addresses these hurdles by providing a robust UDA framework that adapts models to new modalities using only unlabeled target data.

Methodology

The DRL-STNet framework operates through a two-phase pipeline designed to bridge the modality gap while preserving anatomical integrity.

1. Image Translation via Disentangled Representation Learning (DRL)

Traditional translation methods, such as CycleGAN, often struggle to maintain fine anatomical details. DRL-STNet utilizes a Generative Adversarial Network (GAN) to explicitly separate an image into two components: Content (anatomical structure) and Style (modality-specific appearance). By swapping the style of a labeled CT slice with the style of an unlabeled MRI slice, the framework generates "synthetic MRIs." These images possess the visual characteristics of real MRI scans while retaining the precise, ground-truth anatomical labels of the original CT scans.

2. Iterative Self-Training (ST)

To overcome the "reality gap"—where synthetic images may lack the subtle artifacts or textures of clinical scans—the framework employs an iterative self-training loop:

  • Initial Training: A 3D segmentation model (nnU-Net) is trained on the synthetic MRI data.
  • Pseudo-Labeling: The model generates predictions (pseudo-labels) for real, unlabeled MRI scans.
  • Fine-Tuning: The model is iteratively retrained using a combination of synthetic data (with ground-truth labels) and real data (with the generated pseudo-labels). This allows the model to adapt to the specific noise patterns of real-world clinical data.

Experimental Setup

The framework was evaluated using a large-scale, multi-center dataset:

  • Source Data: 2,050 abdominal CT scans from various benchmarks (TCIA, LiTS, MSD).
  • Target Data: Over 4,000 unlabeled MRI scans.
  • Evaluation: Testing was conducted on the FLARE challenge dataset, comprising 110 validation and 300 testing MRI scans across multiple sequences (T1, T2, DWI).

Results and Discussion

DRL-STNet established a new benchmark for cross-modality abdominal organ segmentation. Key performance metrics include:

  • Overall Accuracy: The method outperformed previous state-of-the-art approaches (e.g., SIFA and CycleGAN) by 11.4% in DSC and 13.1% in Normalized Surface Dice (NSD).
  • Organ-Specific Success: The model achieved high accuracy on large organs, such as the Liver (93.76% DSC) and Kidneys (~91% DSC).
  • Clinical Viability: The average inference time is approximately 41 seconds per volume, suggesting potential for integration into clinical workflows.

Despite these gains, challenges remain in segmenting smaller or less distinct structures, such as the adrenal glands (49.78% DSC) and the gallbladder. Furthermore, the quality of the final segmentation remains dependent on the initial GAN-based translation; anatomical artifacts introduced during translation can propagate through the self-training loop.

Conclusion

DRL-STNet demonstrates that combining disentangled representation learning with iterative self-training is a highly effective strategy for unsupervised domain adaptation in medical imaging. By successfully bridging the gap between CT and MRI without the need for manual target labels, this framework paves the way for more scalable and robust medical AI. Future research will explore the use of Diffusion Models for higher-quality translation and multi-task learning to handle broader clinical scenarios.

Summary updated

Abstract

Unsupervised domain adaptation (UDA) is essential for medical image segmentation, especially in cross-modality data scenarios. UDA aims to transfer knowledge from a labeled source domain to an unlabeled target domain, thereby reducing the dependency on extensive manual annotations. This paper presents DRL-STNet, a novel framework for cross-modality medical image segmentation that leverages generative adversarial networks (GANs), disentangled representation learning (DRL), and self-training (ST). Our method leverages DRL within a GAN to translate images from the source to the target modality. Then, the segmentation model is initially trained with these translated images and corresponding source labels and then fine-tuned iteratively using a combination of synthetic and real images with pseudo-labels and real labels. The proposed framework exhibits superior performance in abdominal organ segmentation on the FLARE challenge dataset, surpassing state-of-the-art methods by 11.4% in the Dice similarity coefficient and by 13.1% in the Normalized Surface Dice metric, achieving scores of 74.21% and 80.69%, respectively. The average running time is 41 seconds, and the area under the GPU memory-time curve is 11,292 MB. These results indicate the potential of DRL-STNet for enhancing cross-modality medical image segmentation tasks.
BibTeX
@misc{lin2024drl,
  author = {Hui Lin and Florian Schiffers and Santiago López-Tapia and Neda Tavakoli and Daniel Kim and Aggelos Katsaggelos},
  title = {{DRL-STNet: Unsupervised Domain Adaptation for Cross-modality Medical Image Segmentation via Disentangled Representation Learning}},
  year = {2024},
  eprint = {2409.18340},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.18340},
}
Figure