Abstract
The Segment Anything Model (SAM) has demonstrated remarkable zero-shot generalization in 2D natural image segmentation. However, its application to 3D medical imaging is hindered by the requirement for manual prompting, a lack of semantic labeling, and the inability to capture volumetric spatial dependencies. We present MaskSAM, an adaptation of SAM designed for automated 3D medical image segmentation. MaskSAM introduces an automatic prompt generator and a mask classification head, enabling "prompt-free" operation while assigning anatomical labels. By incorporating lightweight 3D-aware adapters into a frozen SAM backbone, MaskSAM achieves state-of-the-art performance on major benchmarks, including AMOS2022, Synapse, and ACDC, outperforming established models like nnUNet.
Introduction
While SAM's "zero-shot" capabilities are transformative, three primary challenges limit its clinical utility in medical imaging:
- The Prompt Bottleneck: Clinical workflows involving hundreds of 3D slices cannot accommodate manual point or box prompts.
- Semantic Identity: SAM is a binary segmenter; it identifies where an object is but not what it is (e.g., distinguishing a liver from a spleen).
- Dimensionality Gap: SAM's 2D training ignores the critical inter-slice spatial relationships inherent in CT and MRI volumes.
MaskSAM addresses these by transforming SAM into an end-to-end, multi-class 3D segmentation framework.
Methods
The MaskSAM architecture preserves the pre-trained knowledge of SAM while specializing it for medical tasks through three key components:
Automatic Prompt Generation and Classification
Instead of manual input, MaskSAM utilizes a Prompt Generator sub-network. This module extracts multi-level features from the image encoder to automatically produce auxiliary bounding boxes and masks, which serve as prompts for the mask decoder. To provide semantic labels, a classification head with learnable "classifier tokens" is added to the decoder, allowing the model to categorize generated masks into specific anatomical classes.
3D-Aware Adapters
To bridge the 2D-3D gap efficiently, MaskSAM employs Parameter-Efficient Fine-Tuning (PEFT). Rather than retraining the entire model, lightweight 3D Depth-Convolution (DConv) Adapters and 3D Depth-MLP Adapters are inserted into the frozen SAM architecture. These layers learn depth-wise spatial relationships across slices with minimal additional parameters.
Training Strategy
The model is trained using bipartite matching, a technique borrowed from object detection (e.g., DETR). This approach mathematically pairs predicted masks with ground truth labels, facilitating the simultaneous learning of multiple organ classes.
Results and Discussion
MaskSAM was evaluated on three prominent datasets: AMOS2022 (abdominal CT), Synapse (multi-organ), and ACDC (cardiac MRI).
Quantitative Performance
MaskSAM consistently outperformed the industry-standard nnUNet:
- AMOS2022: Achieved a Dice Score of 90.52%, a 2.7% improvement over nnUNet.
- ACDC: Surpassed nnUNet by 1.7%.
- Synapse: Surpassed nnUNet by 1.0%.
Notably, MaskSAM achieves this performance with only 14M tunable parameters, adapting a 91M parameter ViT-base SAM backbone.
Qualitative Analysis
Qualitative results demonstrate that MaskSAM significantly reduces segmentation outliers and accurately identifies organ boundaries through its self-generated prompts.
Conclusion
MaskSAM successfully transitions SAM into the medical domain by automating the prompting process and adding semantic awareness. While it currently relies on a computationally intensive ViT backbone, its superior performance over nnUNet marks a significant step toward general-purpose medical image segmentation models. Future work will focus on optimizing backbone efficiency and extending the framework to multi-modal data.
Summary updated