Enhancing Unsupervised Domain Adaptation via Semantic Similarity Constraint for Medical Image Segmentation

Tao Hu, Shiliang Sun, Jing Zhao, Dongyu ShiIJCAI 2022 · DOI

Introduction

Deep learning models for medical imaging often exhibit significant performance degradation when applied to data from a different imaging modality than the one used during training. This phenomenon, known as domain shift, is particularly prevalent between Magnetic Resonance Imaging (MRI) and Computed Tomography (CT). While both modalities may capture the same anatomical structures, their underlying physical principles result in vastly different pixel intensities and textures, rendering models trained on one modality "brittle" when deployed on the other.

Unsupervised Domain Adaptation (UDA) seeks to bridge this gap without requiring expensive manual labels in the target domain. Traditional UDA methods frequently rely on Generative Adversarial Networks (GANs) to align image styles. However, GANs can be unstable to train and may inadvertently distort or destroy fine-grained anatomical details. This paper proposes a novel framework that introduces a direct semantic constraint to ensure the model learns domain-invariant features—focusing on the shape and logic of organs rather than their superficial appearance.

Methodology

The proposed framework consists of three synergistic components designed to achieve robust cross-modality segmentation:

  1. Image Appearance Transformation: The model utilizes a shallow, shared content encoder ($E_c$) and domain-specific decoders ($U_s, U_t$) to translate images between modalities (e.g., MRI to CT). This process preserves anatomical content through cycle-consistency and adversarial losses.
  2. Semantic Similarity Mining (SSM): This core innovation employs a Siamese-style constraint. The shared encoder processes two different random augmentations of the same image. A cosine similarity loss ($L_{sim}$) is applied to the resulting feature maps, forcing the encoder to ignore visual noise and focus strictly on consistent semantic structures.
  3. Synergistic Segmentation Sub-network: A segmentation network is trained using both real target images and translated source images. It fuses features from the shared content encoder with a dedicated segmentation encoder to leverage both global style alignment and local semantic consistency.
Overview of the proposed UDA framework featuring the Semantic Similarity Mining (SSM) module.

A key design choice is the use of a "shallow" shared encoder (9 layers). The authors argue that a deeper encoder might gain excessive capacity, potentially learning to distinguish between domains rather than identifying the invariant features necessary for successful adaptation.

Experimental Results

The framework was evaluated on two challenging cross-modality benchmarks: Cardiac Substructure Segmentation (MMWHS 2017) and Abdominal Multi-organ Segmentation (CT from Landman et al. and MRI from the ISBI 2019 CHAOS Challenge).

Quantitative Performance

The proposed method achieved state-of-the-art (SOTA) results, significantly outperforming existing methods like SIFA and CyCADA:

  • Cardiac (MR to CT): The model achieved an average Dice score of 80.8%, a notable improvement over previous benchmarks (DSAN at 78.5% and SIFA at 74.1%).
  • Abdominal (MR to CT): The model reached a Dice score of 84.2%, surpassing SIFA's 83.7%.

Ablation studies confirmed the importance of the SSM module; removing it caused the Cardiac Dice score to drop from 80.8% to 76.9%, highlighting the similarity constraint as a primary driver of performance.

Qualitative Analysis

Visual results demonstrate that the model produces cleaner segmentation masks with fewer false-positive artifacts compared to methods relying solely on adversarial alignment. The SSM module ensures that anatomical structures are recognized consistently across the visual distortions inherent in modality shifts.

Qualitative segmentation result of the proposed method on a cardiac CT scan, showing high overlap with anatomical structures.

Conclusion and Future Work

By integrating Semantic Similarity Mining into a UDA framework, this research provides a more stable and effective way to align features across imaging modalities. The direct constraint on the latent space avoids the common pitfalls of GAN-based alignment. Future work includes extending the framework from 2D slices to 3D volumes to fully leverage the volumetric nature of medical data and exploring the use of the SSM module for automated quality assessment of image translations.

Summary updated

BibTeX
@inproceedings{hu2022enhancing,
  author = {Tao Hu and Shiliang Sun and Jing Zhao and Dongyu Shi},
  title = {{Enhancing Unsupervised Domain Adaptation via Semantic Similarity Constraint for Medical Image Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/426},
}
Figure