Cross-model Mutual Learning for Exemplar-based Medical Image Segmentation

Qing En, Yuhong Guo

Abstract

Medical image segmentation typically demands extensive dense annotations for model training, which is both time-consuming and skill-intensive. To mitigate this burden, exemplar-based medical image segmentation methods have been introduced to achieve effective training with only one annotated image. In this paper, we introduce a novel Cross-model Mutual learning framework for Exemplar-based Medical image Segmentation (CMEMS), which leverages two models to mutually excavate implicit information from unlabeled data at multiple granularities. CMEMS can eliminate confirmation bias and enable collaborative training to learn complementary information by enforcing consistency at different granularities across models. Concretely, cross-model image perturbation based mutual learning is devised by using weakly perturbed images to generate high-confidence pseudo-labels, supervising predictions of strongly perturbed images across models. This approach enables joint pursuit of prediction consistency at the image granularity. Moreover, cross-model multi-level feature perturbation based mutual learning is designed by letting pseudo-labels supervise predictions from perturbed multi-level features with different resolutions, which can broaden the perturbation space and enhance the robustness of our framework. CMEMS is jointly trained using exemplar data, synthetic data, and unlabeled data in an end-to-end manner. Experimental results on two medical image datasets indicate that the proposed CMEMS outperforms the state-of-the-art segmentation methods with extremely limited supervision.

Introduction

Deep learning has revolutionized medical image analysis, but its heavy reliance on large-scale, expert-annotated datasets remains a significant bottleneck. Pixel-perfect annotations for CT or MRI scans require substantial time from highly trained radiologists. To lower this barrier, researchers have explored "exemplar-based" learning—an extreme scenario where a model is trained using only a single annotated image and a large pool of unlabeled data.

The primary challenge in this setting is confirmation bias. When a single model generates its own pseudo-labels for unlabeled data to train itself, it tends to reinforce its own initial mistakes. Furthermore, existing methods often fail to exploit the multi-scale nature of medical images, ignoring how information is represented at different resolutions (granularities) within a neural network.

Conceptual overview of the exemplar learning scenario and the mutual learning process between two segmentation networks.

Methodology

The proposed CMEMS framework is designed to excavate implicit information from unlabeled data at multiple granularities through several novel strategies.

Dual-Model Mutual Learning

Instead of a single model, CMEMS utilizes two identical networks (e.g., U-Nets) with different initializations. They work collaboratively to "excavate" information from unlabeled data, preventing the models from simply memorizing their own errors and thus mitigating confirmation bias.

Cross-Model Image Perturbation

Model A processes a "weakly" perturbed image (e.g., simple rotation or flipping) to generate high-confidence pseudo-labels. These labels are then used to supervise Model B, which processes a "strongly" perturbed version (e.g., color jittering) of the same image. This teacher-student swap occurs in both directions, ensuring joint pursuit of prediction consistency at the image granularity.

Multi-Level Feature Perturbation (MLFP)

To enhance robustness, the framework applies perturbations—such as randomly dropping feature channels—to internal layers of the network at different resolutions. The models must maintain consistency between these noisy internal features and the pseudo-labels, broadening the perturbation space and forcing the models to learn underlying anatomical structures.

Detailed architecture of the CMEMS framework, illustrating labeled data paths, image perturbation paths, and feature perturbation paths across two networks.

Experiments and Results

The framework was evaluated on two standard medical imaging benchmarks: the Synapse Multi-organ CT dataset and the ACDC (Automated Cardiac Diagnosis Challenge) MRI dataset. In both experiments, the models were restricted to seeing the labels of only one image during training.

CMEMS significantly outperformed previous state-of-the-art (SOTA) methods like ELSNet:

  • Synapse Dataset: CMEMS achieved a 0.597 Dice Similarity Coefficient (DSC), nearly doubling the performance of the previous SOTA (0.315). It also drastically reduced the average Hausdorff Distance (HD95) from 109.70 to 55.02.
  • ACDC Dataset: Performance reached a 0.817 DSC. For context, a fully supervised model trained on the entire dataset achieves approximately 0.898, meaning CMEMS reached near-expert levels using only a single label.
  • Small Organ Performance: The model showed remarkable improvements in segmenting difficult, small organs. For instance, the DSC for the Aorta jumped from 0.319 to 0.840.

Conclusion

CMEMS demonstrates that cross-model mutual learning and multi-granularity perturbations can effectively overcome the limitations of exemplar-based segmentation. By breaking the cycle of confirmation bias and enforcing consistency across different scales, the framework achieves state-of-the-art performance with extremely limited supervision. Future research may extend this framework to 3D medical image segmentation or cross-modality adaptation tasks.

Summary updated

Abstract

Medical image segmentation typically demands extensive dense annotations for model training, which is both time-consuming and skill-intensive. To mitigate this burden, exemplar-based medical image segmentation methods have been introduced to achieve effective training with only one annotated image. In this paper, we introduce a novel Cross-model Mutual learning framework for Exemplar-based Medical image Segmentation (CMEMS), which leverages two models to mutually excavate implicit information from unlabeled data at multiple granularities. CMEMS can eliminate confirmation bias and enable collaborative training to learn complementary information by enforcing consistency at different granularities across models. Concretely, cross-model image perturbation based mutual learning is devised by using weakly perturbed images to generate high-confidence pseudo-labels, supervising predictions of strongly perturbed images across models. This approach enables joint pursuit of prediction consistency at the image granularity. Moreover, cross-model multi-level feature perturbation based mutual learning is designed by letting pseudo-labels supervise predictions from perturbed multi-level features with different resolutions, which can broaden the perturbation space and enhance the robustness of our framework. CMEMS is jointly trained using exemplar data, synthetic data, and unlabeled data in an end-to-end manner. Experimental results on two medical image datasets indicate that the proposed CMEMS outperforms the state-of-the-art segmentation methods with extremely limited supervision.
BibTeX
@inproceedings{en2024cross,
  author = {Qing En and Yuhong Guo},
  title = {{Cross-model Mutual Learning for Exemplar-based Medical Image Segmentation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=nd8ysYgl7M},
}
Figure