MaskSAM: Towards Auto-prompt SAM with Mask Classification for Medical Image Segmentation

Bin Xie, Hao Tang, Bin Duan, Dawen Cai, Yan Yan, Gady AgamSubmission to ICLR 2025 · Withdrawn

TL;DR

While Meta's Segment Anything Model (SAM) excels at prompt-guided 2D natural image segmentation, it struggles with 3D multi-organ medical scans that lack manual prompts and require multi-class semantic labels. MaskSAM adapts SAM into a fully automated, prompt-free 3D segmentation network by introducing an internal prompt generator, mask-classification tokens, and volumetric adapters—achieving state-of-the-art results across AMOS2022, Synapse, and ACDC benchmarks without altering SAM’s core pre-trained weights.

What Problem Does This Paper Address?

Applying SAM directly to clinical workflows faces three fundamental roadblocks:

  1. Prompt Dependency: SAM requires user-supplied points or bounding boxes to localize targets. In automated clinical pipelines and segmentation challenges, ground-truth prompts are unavailable at inference.
  2. Lack of Semantic Labels: SAM predicts binary foreground/background masks per prompt, but medical tasks demand assigning distinct semantic identities (e.g., distinguishing spleen, liver, and kidneys) across multiple co-occurring organs.
  3. The 2D-to-3D Domain Gap: Standard medical modalities (CT, MRI) are volumetric 3D scans with inter-slice continuity, whereas SAM operates solely on 2D natural RGB images.

Previous medical adaptations often discarded SAM's prompt encoder or mask decoder to build standard U-Net-like networks, sacrificing the rich priors gained from training on over 1 billion masks. MaskSAM preserves SAM's full architecture intact, freezing its pre-trained weights while introducing lightweight modules to solve prompt generation, mask classification, and 3D feature extraction.

Methods

MaskSAM keeps the entire SAM model frozen and introduces three key mechanisms:

  • Automated Prompt Generator: Instead of relying on manual inputs, this module extracts multi-level feature maps from the frozen Vision Transformer (ViT) image encoder. It upsamples and combines them to generate $N$ auxiliary binary masks, bounding boxes, and classifier tokens. The best-performing setup averages bounding boxes derived directly from the predicted masks with learned box queries.
  • Mask Classification via Bipartite Matching: Inspired by DETR and MaskFormer, MaskSAM pairs each predicted binary mask with a class probability distribution. The mask decoder combines auxiliary classifier tokens from the prompt generator with learnable global classifier tokens. The network matches predictions to variable numbers of ground-truth organs using bipartite Hungarian matching and trains end-to-end with composite classification, box regression, and Dice/cross-entropy losses.
  • 3D Volumetric Adapters:
  • DConvAdapter (3D Depth-Convolution Adapter): Injected into every transformer block of the image encoder and mask decoder image-attention paths. It sandwiches a 3D depth-wise convolution and skip connection between down- and up-projection layers to capture inter-slice volumetric geometry.
  • DMLPAdapter (3D Depth-MLP Adapter): Injected into the mask decoder's prompt-attention blocks, using multi-layer perceptrons over the depth dimension to model slice-to-slice relationships for prompt tokens.
  • Volumetric Preprocessing: A convolutional bottleneck maps varying input modalities/channels to 3 channels, accompanied by learnable depth positional embeddings.
Raw 3D Scan ──> [Modality Adapter + ViT Encoder (w/ DConvAdapters)] ──> Image Embeddings
                         │
                         ├──> [Prompt Generator] ──> Auxiliary Masks, Boxes & Class Tokens
                         │                                │
                         ▼                                ▼
              [Mask Decoder (w/ DConv & DMLP Adapters)] + [Global Class Tokens]
                                         │
                                         ▼
                     Bipartite Matching & Multi-Class Predictions

Datasets

  • SA-1B: Pre-training source for the base 2D SAM model.
  • AMOS22 (Abdominal CT): 200 training and 200 testing CT scans spanning 16 abdominal organs; evaluated under 5-fold cross-validation.
  • Synapse Multi-Organ Segmentation: 30 abdominal CT scans (18 train, 12 validation) evaluated across 8 organs.
  • ACDC (Cardiac MRI): 100 patient cases (70 train, 10 validation, 20 test) evaluating segmentation of the right ventricle, myocardium, and left ventricle.

Key Insights and Ideas

  • Preserving SAM’s integrity beats re-architecting it: Rather than stripping off the prompt encoder or decoder, feeding SAM internally generated pseudo-prompts (masks + boxes) allows the model to leverage its native interactive design without human intervention.
  • Hybrid box generation provides stability: Deriving bounding boxes solely from learned queries or solely from auxiliary masks produces inferior results. Averaging the two provides a self-consistent prior that stabilizes bounding box coordinates.
  • Asymmetric adapters match token types: Spatial image tokens benefit heavily from localized 3D depth-wise convolutions, whereas prompt tokens—which lack strict 2D spatial grid layouts—benefit more from cross-depth MLP interactions.

Key Results

  • AMOS2022: MaskSAM achieved a state-of-the-art 90.52% Dice Similarity Coefficient (DSC), outperforming the benchmark standard nnUNet (87.8%) by 2.7%, SwinUNETR by 2.5%, and outperforming previous SAM adaptations like SAMed (77.3%) and SAM3D (75.2%) by over 13–15%.
  • ACDC: Reached 93.39% average DSC, beating nnFormer (92.06%) by 1.3% and nnUNet (91.61%) by 1.7%.
  • Synapse: Attained 87.23% DSC, surpassing nnFormer (86.57%) and nnUNet (86.21%), while substantially exceeding prompt-reliant MedSAM (64.65% with bounding boxes).

Limitations and Future Work

  • No Dedicated Discussion in Paper: The authors do not include an explicit limitations or future work section.
  • Inherent Limitations:
  • The architecture relies on an upper bound $N$ for the maximum number of object instances/classes per volume, which must be set per dataset.
  • Evaluating 3D volumes through a high-resolution ViT backbone—even with parameter-efficient fine-tuning—incurs notable GPU memory overhead (experiments required 40GB NVIDIA A6000 GPUs).
  • Future directions could explore extension to larger multi-modal datasets, open-vocabulary organ segmentation, or deploying smaller ViT variants (e.g., ViT-Tiny/Small) for resource-constrained clinical settings.

Summary updated

Abstract

Segment Anything Model~(SAM), a prompt-driven foundation model for natural image segmentation, demonstrated impressive zero-shot performance. However, SAM does not work when directly applied to medical image segmentation tasks, since SAM lacks the functionality to predict semantic labels for predicted masks and needs to provide extra prompts, such as points or boxes, to segment target regions. Meanwhile, there is a significant gap between 2D natural images and 3D medical images, so the performance of SAM is imperfect for medical image segmentation tasks. Following the above issues, we propose MaskSAM, a novel mask classification prompt-free SAM adaptation framework for medical image segmentation. We design a prompt generator combined with the image encoder in SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks, and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts, which addresses the requirements of extra prompts, is associated with class label predictions by the sum of the auxiliary classifier token and the learnable global classifier tokens in the mask decoder of SAM to solve the predictions of semantic labels. Meanwhile, we design a 3D depth-convolution adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings. We inject one of them into each transformer block in the image encoder and mask decoder to enable pre-trained 2D SAM models to extract 3D information and adapt to 3D medical images. Our method achieves state-of-the-art performance on AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.
BibTeX
@misc{xie2024masksam,
  author = {Bin Xie and Hao Tang and Bin Duan and Dawen Cai and Yan Yan and Gady Agam},
  title = {{MaskSAM: Towards Auto-prompt SAM with Mask Classification for Medical Image Segmentation}},
  year = {2024},
  eprint = {2403.14103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.14103},
}
Figure