Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation

Cheng Chen, Qi Dou, Hao Chen, Jing Qin, Pheng-Ann Heng

Introduction

Deep learning has revolutionized medical image segmentation, yet these models often struggle with "domain shift." A network trained to high accuracy on MRI scans frequently fails when applied to CT scans due to differing visual characteristics, despite the underlying anatomy being identical. While manual annotation of new modalities could solve this, it is prohibitively expensive and requires expert radiologists.

This paper introduces SIFA (Synergistic Image and Feature Alignment), an Unsupervised Domain Adaptation (UDA) framework. SIFA aims to bridge the gap between a labeled source domain (e.g., MRI) and an unlabeled target domain (e.g., CT) by aligning them at both the image and feature levels simultaneously.

The SIFA Framework

The core innovation of SIFA is the "synergy" between image-level and feature-level alignment. Unlike previous methods that treated these as sequential steps, SIFA utilizes a shared encoder to unify these objectives, allowing them to provide complementary regularizations.

The SIFA architecture integrates image and feature alignment through a shared encoder.

Synergistic Image and Feature Alignment

SIFA employs several key components to achieve robust adaptation:

  1. Image Alignment: Using a CycleGAN-style architecture, the model transforms source images into the visual style of the target domain. This addresses low-level appearance differences (e.g., intensity distributions).
  2. Feature Alignment: To handle residual domain shift that pixel-level transformations cannot fix, SIFA enforces domain-invariance in "compact" spaces. Specifically, it aligns features in the semantic prediction space (the final segmentation masks) and the generated image space. Since anatomy is consistent across modalities, the predicted shapes should be indistinguishable regardless of the input domain.
  3. Deeply Supervised Alignment: Discriminators are connected to multiple layers of the shared encoder. This ensures that both high-level semantic information and low-level textural features are aligned across domains.
  4. Shared Encoder: The encoder $E$ is shared across transformation, reconstruction, and segmentation tasks. This ensures the learned features are robust and generalizable across both domains.

Experimental Results

The framework was evaluated on two challenging bidirectional tasks: Cardiac Substructure Segmentation (MMWHS 2017) and Abdominal Multi-organ Segmentation (ISBI 2019 CHAOS vs. public CT data).

Quantitative Performance

SIFA demonstrated significant improvements over baseline models and outperformed previous state-of-the-art methods like CyCADA:

  • Cardiac (MRI $\to$ CT): Without adaptation, the model achieved a Dice score of only 17.2%. SIFA boosted this to 74.1%, surpassing the previous best of 73.0%.
  • Abdominal (CT $\to$ MRI): SIFA reached an average Dice of 85.4%. This is remarkably close to the "Upper Bound" of 87.3% (a model trained with full labels on the target data), meaning SIFA recovered nearly all performance lost to domain shift without using any target labels.

Qualitative Analysis

Visual results confirm the quantitative findings. While models without adaptation produce fragmented and noisy masks, SIFA generates clean, anatomically correct structures that closely mirror the ground truth.

Visual comparison of segmentation performance on cardiac and abdominal datasets.

Conclusion

SIFA effectively addresses the domain shift problem in medical imaging by leveraging the synergy between image and feature alignment. By forcing the network to learn domain-invariant features in compact spaces through a shared encoder, the framework achieves state-of-the-art performance in unsupervised cross-modality segmentation. Future work may involve extending this 2D slice-based approach to a full 3D framework to better capture volumetric context.

Summary updated

BibTeX
@article{chen2020unsupervised,
  author = {Cheng Chen and Qi Dou and Hao Chen and Jing Qin and Pheng-Ann Heng},
  title = {{Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2020},
  doi = {10.1109/tmi.2020.2972701},
}
Figure