Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration

Xiaogen Zhon, Yiyou Sun, Min Deng, Winnie Chiu Wing Chu, Qi Dou

Introduction

Multimodal learning, which integrates data from different imaging sources like CT and MRI, is a cornerstone of accurate medical image segmentation. However, two primary challenges hinder its clinical adoption: the high cost of expert-labeled data and the frequent anatomical misalignment between different modalities due to patient movement or varying resolutions.

This research introduces a novel semi-supervised framework designed to be robust under these imperfect conditions. By leveraging large amounts of unlabeled data and focusing on feature-level rather than pixel-level alignment, the system maintains high accuracy even when spatial shifts occur between modalities.

Proposed Methodology

The framework is built upon three core pillars designed to maximize information extraction from limited labels and misaligned scans:

  1. 3D Foundation Model Encoders: The architecture utilizes pre-trained 3D encoders (based on SAM-Med3D) with specialized adapters. These adapters allow for efficient fine-tuning on medical tasks without the need to retrain the entire backbone network.
  2. Cross Modality Collaboration (CMC): A Modality-Independent Awareness (MIA) module is employed to distill core anatomical features that remain consistent across different imaging types (e.g., identifying a liver regardless of whether it is viewed via CT or MRI).
  3. Dual Consistency Losses:
    • Channel-wise Semantic Consistency (CSC) Loss: This aligns features in the latent space, making the model resilient to spatial misalignments.
  4. Contrastive Anatomical-similar Consistency (CAC) Loss: This forces the model to ensure that predicted structures from one modality match those from another in unlabeled data.
Figure 1: Overview of the proposed framework, highlighting the 3D modality-specific encoders, Cross Modality Collaboration module, and the dual consistency learning strategy.

Experimental Validation

The framework was validated across three diverse and challenging datasets:

  • MS-CMRSeg: Registered cardiac MRI images (LGE, T2w, and BSSFP).
  • AMOS: An abdominal multi-organ dataset featuring unpaired CT and MRI scans from different patients, providing a rigorous test for misalignment robustness.
  • TAO: A private dataset for Thyroid-Associated Orbitopathy with unpaired T1 and T1c MRI scans.

Results and Discussion

The proposed method consistently outperformed state-of-the-art (SOTA) multimodal methods, particularly in label-scarce scenarios. On the AMOS dataset using only 10% of labels, the model achieved Dice scores of 76.3% for CT and 84.3% for MRI, surpassing previous benchmarks like CML and mmFor.

Figure 2: Performance comparison (Dice score %) on the MS-CMRSeg dataset for BSSFP and LGE modalities across different anatomical structures (MYO, RV, LV).

Key insights from the results include:

  • Feature-wise Alignment: Focusing on the "meaning" of features rather than exact spatial coordinates allows the model to correctly identify organs even when they are shifted by several centimeters between scans.
  • Synergy in Semi-Supervision: The use of unlabeled data with contrastive learning effectively regularizes anatomical structures across modalities, proving that unlabeled data is a vital component for robust segmentation.

Conclusion

This framework represents a significant step toward practical multimodal medical imaging. By addressing the "data hunger" and misalignment problems, it offers a more resilient solution for clinical environments. Future work will focus on optimizing the computational load of 3D foundation models and exploring scenarios where one imaging modality may be entirely missing.

Summary updated

BibTeX
@inproceedings{zhon2024robust,
  author = {Xiaogen Zhon and Yiyou Sun and Min Deng and Winnie Chiu Wing Chu and Qi Dou},
  title = {{Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  doi = {10.1007/978-3-031-72378-0\_6},
}

Loading related content...

Figure