AdvMIM: Adversarial Masked Image Modeling for Semi-Supervised Medical Image Segmentation

Lei Zhu, Jun Zhou, Rick Siow Mong Goh, Yong Liu

TL;DR

This paper introduces AdvMIM, a novel semi-supervised learning method for medical image segmentation that leverages Vision Transformers (ViTs). By creating an "auxiliary masked domain" and employing adversarial training, AdvMIM significantly boosts segmentation performance with limited labeled data, addressing a key bottleneck in applying ViTs to medical imaging.

What Problem Does This Paper Address?

Medical image segmentation is crucial for diagnosis and treatment planning. Vision Transformers (ViTs) have shown great promise due to their ability to capture long-range dependencies, but they are notoriously data-hungry. This makes them challenging to use in semi-supervised learning scenarios for medical imaging, where obtaining large amounts of expertly annotated data is expensive and time-consuming. Existing semi-supervised methods, even those combining CNNs and Transformers, still struggle to effectively train ViTs with limited labels. The core problem is the insufficient supervision signal available for training these powerful models in low-data regimes.

Methods

The proposed method, AdvMIM (Adversarial Masked Image Modeling), tackles the supervision signal scarcity by introducing two main innovations:

  • Masked Domain Learning: Instead of the typical masked image modeling task of reconstructing image patches, AdvMIM trains the transformer to predict the entire segmentation mask from masked input images. This forces the model to learn richer semantic representations from the visible parts of the image, effectively creating an auxiliary "masked domain" that provides additional supervision. This masked domain is trained using both original labels from labeled data and pseudo-labels generated from unlabeled data.
  • Adversarial Masked Domain Adaptation: To further enhance learning, AdvMIM employs an adversarial training strategy. A domain discriminator is introduced to distinguish between segmentation masks generated from the original (labeled) domain and the masked (unlabeled) domain. The transformer is then trained to "fool" this discriminator, effectively minimizing the domain gap between the original and masked data distributions. This process is theoretically grounded in multi-domain learning, showing that reducing this gap is crucial for improving performance.

The authors also extend this AdvMIM approach to CNNs, allowing for a combined training strategy.

Datasets

The paper evaluates AdvMIM on three publicly available medical image segmentation datasets:

  • Automated Cardiac Diagnosis Challenge (ACDC): Contains 100 MRI scans of three organs.
  • Synapse: Consists of 30 CT scans with eight abdominal organs annotated.
  • International Skin Imaging Collaboration (ISIC): A skin lesion segmentation dataset with 2,594 dermoscopy images.

The experiments utilize various low-label data partitions (e.g., 3%, 10%, 15%, 30%) to simulate semi-supervised scenarios.

Key Insights and Ideas

  • Masked Image Modeling for Segmentation Supervision: The core idea of adapting masked image modeling (MIM) from representation learning to directly boost segmentation performance is a clever way to generate more supervision signals for ViTs. Predicting the whole mask from partial input is a more challenging and informative task than just reconstructing patches.
  • Domain Gap Hinders Semi-Supervised Learning: The theoretical analysis and experimental results strongly suggest that a significant domain gap between the original and auxiliary (masked) domains can negatively impact semi-supervised learning performance. This highlights the importance of domain adaptation techniques.
  • Adversarial Training as a Domain Aligner: Using adversarial training to explicitly minimize the domain gap between the original and masked data distributions is a powerful mechanism to improve the robustness and effectiveness of the learned representations.
  • ViTs Benefit Greatly from Extra Supervision: The ablation studies clearly demonstrate that each component of AdvMIM (weighted cross-teaching, masked domain learning, and adversarial adaptation) contributes to performance gains, underscoring the need for more supervision signals for ViTs in low-data settings.

Key Results

AdvMIM achieves state-of-the-art results across all tested datasets and label partitions.

  • ACDC (3% labeled data): Outperforms the previous best method (M-CnT) by a significant margin of 10.1% in Dice score and 8.7mm in HD.
  • Synapse (15% labeled data): Achieves 1.0% higher Dice and 9.9mm lower HD compared to the previous best.
  • ISIC (3% labeled data): Shows a tremendous improvement in HD score (9.9mm lower) and a 1.9% higher Dice score.
  • The method demonstrates remarkable applicability even with extremely limited annotations, achieving high performance (e.g., 85.4% Dice on ACDC with 3% labels).

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of transformers and adversarial training generally implies higher computational requirements compared to simpler CNN-based methods.
  • Hyperparameter Sensitivity: The paper includes a sensitivity analysis for the adversarial weight ($\lambda_{adv}$), showing that while robust within a range, excessively large values can degrade performance. This suggests careful tuning is still necessary.
  • Future Work: The paper mentions extending AdvMIM to CNN networks, which was done and showed further improvement. The success of this method opens avenues for exploring more sophisticated masked domain generation strategies or different adversarial objectives for semi-supervised medical image segmentation. The theoretical insights into domain gap reduction could also inspire new approaches in other domain adaptation tasks.

Summary updated

Abstract

Vision Transformer has recently gained tremendous popularity in medical image segmentation task due to its superior capability in capturing long-range dependencies. However, transformer requires a large amount of labeled data to be effective, which hinders its applicability in annotation scarce semi-supervised learning scenario where only limited labeled data is available. State-of-the-art semi-supervised learning methods propose combinatorial CNN-Transformer learning to cross teach a transformer with a convolutional neural network, which achieves promising results. However, it remains a challenging task to effectively train the transformer with limited labeled data. In this paper, we propose an adversarial masked image modeling method to fully unleash the potential of transformer for semi-supervised medical image segmentation. The key challenge in semi-supervised learning with transformer lies in the lack of sufficient supervision signal. To this end, we propose to construct an auxiliary masked domain from original domain with masked image modeling and train the transformer to predict the entire segmentation mask with masked inputs to increase supervision signal. We leverage the original labels from labeled data and pseudo-labels from unlabeled data to learn the masked domain. To further benefit the original domain from masked domain, we provide a theoretical analysis of our method from a multi-domain learning perspective and devise a novel adversarial training loss to reduce the domain gap between the original and masked domain, which boosts semi-supervised learning performance. We also extend adversarial masked image modeling to CNN network. Extensive experiments on three public medical image segmentation datasets demonstrate the effectiveness of our method, where our method outperforms existing methods significantly. Our code is publicly available at https://github.com/zlheui/AdvMIM.
BibTeX
@inproceedings{zhu2025advmim,
  author = {Lei Zhu and Jun Zhou and Rick Siow Mong Goh and Yong Liu},
  title = {{AdvMIM: Adversarial Masked Image Modeling for Semi-Supervised Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2025},
  doi = {10.1007/978-3-032-05325-1\_6},
}
Figure