TL;DR
While Meta's Segment Anything Model (SAM) excels at prompt-guided 2D natural image segmentation, it struggles with 3D multi-organ medical scans that lack manual prompts and require multi-class semantic labels. MaskSAM adapts SAM into a fully automated, prompt-free 3D segmentation network by introducing an internal prompt generator, mask-classification tokens, and volumetric adapters—achieving state-of-the-art results across AMOS2022, Synapse, and ACDC benchmarks without altering SAM’s core pre-trained weights.
What Problem Does This Paper Address?
Applying SAM directly to clinical workflows faces three fundamental roadblocks:
- Prompt Dependency: SAM requires user-supplied points or bounding boxes to localize targets. In automated clinical pipelines and segmentation challenges, ground-truth prompts are unavailable at inference.
- Lack of Semantic Labels: SAM predicts binary foreground/background masks per prompt, but medical tasks demand assigning distinct semantic identities (e.g., distinguishing spleen, liver, and kidneys) across multiple co-occurring organs.
- The 2D-to-3D Domain Gap: Standard medical modalities (CT, MRI) are volumetric 3D scans with inter-slice continuity, whereas SAM operates solely on 2D natural RGB images.
Previous medical adaptations often discarded SAM's prompt encoder or mask decoder to build standard U-Net-like networks, sacrificing the rich priors gained from training on over 1 billion masks. MaskSAM preserves SAM's full architecture intact, freezing its pre-trained weights while introducing lightweight modules to solve prompt generation, mask classification, and 3D feature extraction.
Methods
MaskSAM keeps the entire SAM model frozen and introduces three key mechanisms:
- Automated Prompt Generator: Instead of relying on manual inputs, this module extracts multi-level feature maps from the frozen Vision Transformer (ViT) image encoder. It upsamples and combines them to generate $N$ auxiliary binary masks, bounding boxes, and classifier tokens. The best-performing setup averages bounding boxes derived directly from the predicted masks with learned box queries.
- Mask Classification via Bipartite Matching: Inspired by DETR and MaskFormer, MaskSAM pairs each predicted binary mask with a class probability distribution. The mask decoder combines auxiliary classifier tokens from the prompt generator with learnable global classifier tokens. The network matches predictions to variable numbers of ground-truth organs using bipartite Hungarian matching and trains end-to-end with composite classification, box regression, and Dice/cross-entropy losses.
- 3D Volumetric Adapters:
- DConvAdapter (3D Depth-Convolution Adapter): Injected into every transformer block of the image encoder and mask decoder image-attention paths. It sandwiches a 3D depth-wise convolution and skip connection between down- and up-projection layers to capture inter-slice volumetric geometry.
- DMLPAdapter (3D Depth-MLP Adapter): Injected into the mask decoder's prompt-attention blocks, using multi-layer perceptrons over the depth dimension to model slice-to-slice relationships for prompt tokens.
- Volumetric Preprocessing: A convolutional bottleneck maps varying input modalities/channels to 3 channels, accompanied by learnable depth positional embeddings.
Raw 3D Scan ──> [Modality Adapter + ViT Encoder (w/ DConvAdapters)] ──> Image Embeddings
│
├──> [Prompt Generator] ──> Auxiliary Masks, Boxes & Class Tokens
│ │
▼ ▼
[Mask Decoder (w/ DConv & DMLP Adapters)] + [Global Class Tokens]
│
▼
Bipartite Matching & Multi-Class Predictions
Datasets
- SA-1B: Pre-training source for the base 2D SAM model.
- AMOS22 (Abdominal CT): 200 training and 200 testing CT scans spanning 16 abdominal organs; evaluated under 5-fold cross-validation.
- Synapse Multi-Organ Segmentation: 30 abdominal CT scans (18 train, 12 validation) evaluated across 8 organs.
- ACDC (Cardiac MRI): 100 patient cases (70 train, 10 validation, 20 test) evaluating segmentation of the right ventricle, myocardium, and left ventricle.
Key Insights and Ideas
- Preserving SAM’s integrity beats re-architecting it: Rather than stripping off the prompt encoder or decoder, feeding SAM internally generated pseudo-prompts (masks + boxes) allows the model to leverage its native interactive design without human intervention.
- Hybrid box generation provides stability: Deriving bounding boxes solely from learned queries or solely from auxiliary masks produces inferior results. Averaging the two provides a self-consistent prior that stabilizes bounding box coordinates.
- Asymmetric adapters match token types: Spatial image tokens benefit heavily from localized 3D depth-wise convolutions, whereas prompt tokens—which lack strict 2D spatial grid layouts—benefit more from cross-depth MLP interactions.
Key Results
- AMOS2022: MaskSAM achieved a state-of-the-art 90.52% Dice Similarity Coefficient (DSC), outperforming the benchmark standard nnUNet (87.8%) by 2.7%, SwinUNETR by 2.5%, and outperforming previous SAM adaptations like SAMed (77.3%) and SAM3D (75.2%) by over 13–15%.
- ACDC: Reached 93.39% average DSC, beating nnFormer (92.06%) by 1.3% and nnUNet (91.61%) by 1.7%.
- Synapse: Attained 87.23% DSC, surpassing nnFormer (86.57%) and nnUNet (86.21%), while substantially exceeding prompt-reliant MedSAM (64.65% with bounding boxes).
Limitations and Future Work
- No Dedicated Discussion in Paper: The authors do not include an explicit limitations or future work section.
- Inherent Limitations:
- The architecture relies on an upper bound $N$ for the maximum number of object instances/classes per volume, which must be set per dataset.
- Evaluating 3D volumes through a high-resolution ViT backbone—even with parameter-efficient fine-tuning—incurs notable GPU memory overhead (experiments required 40GB NVIDIA A6000 GPUs).
- Future directions could explore extension to larger multi-modal datasets, open-vocabulary organ segmentation, or deploying smaller ViT variants (e.g., ViT-Tiny/Small) for resource-constrained clinical settings.
Summary updated