MaskSAM: Towards Auto-prompt SAM with Mask Classification for Medical Image Segmentation

Bin Xie, Hao Tang, Bin Duan, Dawen Cai, Yan Yan

Abstract

The Segment Anything Model (SAM) has demonstrated remarkable zero-shot generalization in 2D natural image segmentation. However, its application to 3D medical imaging is hindered by the requirement for manual prompting, a lack of semantic labeling, and the inability to capture volumetric spatial dependencies. We present MaskSAM, an adaptation of SAM designed for automated 3D medical image segmentation. MaskSAM introduces an automatic prompt generator and a mask classification head, enabling "prompt-free" operation while assigning anatomical labels. By incorporating lightweight 3D-aware adapters into a frozen SAM backbone, MaskSAM achieves state-of-the-art performance on major benchmarks, including AMOS2022, Synapse, and ACDC, outperforming established models like nnUNet.

Introduction

While SAM's "zero-shot" capabilities are transformative, three primary challenges limit its clinical utility in medical imaging:

  1. The Prompt Bottleneck: Clinical workflows involving hundreds of 3D slices cannot accommodate manual point or box prompts.
  2. Semantic Identity: SAM is a binary segmenter; it identifies where an object is but not what it is (e.g., distinguishing a liver from a spleen).
  3. Dimensionality Gap: SAM's 2D training ignores the critical inter-slice spatial relationships inherent in CT and MRI volumes.

MaskSAM addresses these by transforming SAM into an end-to-end, multi-class 3D segmentation framework.

Methods

The MaskSAM architecture preserves the pre-trained knowledge of SAM while specializing it for medical tasks through three key components:

Automatic Prompt Generation and Classification

Instead of manual input, MaskSAM utilizes a Prompt Generator sub-network. This module extracts multi-level features from the image encoder to automatically produce auxiliary bounding boxes and masks, which serve as prompts for the mask decoder. To provide semantic labels, a classification head with learnable "classifier tokens" is added to the decoder, allowing the model to categorize generated masks into specific anatomical classes.

The MaskSAM framework overview, illustrating the integration of the Prompt Generator, Modified Mask Decoder with classifier tokens, and the bipartite matching strategy for training.

3D-Aware Adapters

To bridge the 2D-3D gap efficiently, MaskSAM employs Parameter-Efficient Fine-Tuning (PEFT). Rather than retraining the entire model, lightweight 3D Depth-Convolution (DConv) Adapters and 3D Depth-MLP Adapters are inserted into the frozen SAM architecture. These layers learn depth-wise spatial relationships across slices with minimal additional parameters.

Training Strategy

The model is trained using bipartite matching, a technique borrowed from object detection (e.g., DETR). This approach mathematically pairs predicted masks with ground truth labels, facilitating the simultaneous learning of multiple organ classes.

Results and Discussion

MaskSAM was evaluated on three prominent datasets: AMOS2022 (abdominal CT), Synapse (multi-organ), and ACDC (cardiac MRI).

Quantitative Performance

MaskSAM consistently outperformed the industry-standard nnUNet:

  • AMOS2022: Achieved a Dice Score of 90.52%, a 2.7% improvement over nnUNet.
  • ACDC: Surpassed nnUNet by 1.7%.
  • Synapse: Surpassed nnUNet by 1.0%.

Notably, MaskSAM achieves this performance with only 14M tunable parameters, adapting a 91M parameter ViT-base SAM backbone.

Qualitative Analysis

Qualitative results demonstrate that MaskSAM significantly reduces segmentation outliers and accurately identifies organ boundaries through its self-generated prompts.

Qualitative comparison between Ground Truth (GT) and MaskSAM predictions (Pred) across various CT slices, showing the effectiveness of the predicted bounding boxes and semantic masks.

Conclusion

MaskSAM successfully transitions SAM into the medical domain by automating the prompting process and adding semantic awareness. While it currently relies on a computationally intensive ViT backbone, its superior performance over nnUNet marks a significant step toward general-purpose medical image segmentation models. Future work will focus on optimizing backbone efficiency and extending the framework to multi-modal data.

Summary updated

Abstract

Segment Anything Model~(SAM), a prompt-driven foundation model for natural image segmentation, has demonstrated impressive zero-shot performance. However, SAM does not work when directly applied to medical image segmentation tasks, since SAM lacks the functionality to predict semantic labels for predicted masks and needs to provide extra prompts, such as points or boxes, to segment target regions. Meanwhile, there is a huge gap between 2D natural images and 3D medical images, so the performance of SAM is imperfect for medical image segmentation tasks. Following the above issues, we propose MaskSAM, a novel mask classification prompt-free SAM adaptation framework for medical image segmentation. We design a prompt generator combined with the image encoder in SAM to generate a set of auxiliary classifier tokens, auxiliary binary masks, and auxiliary bounding boxes. Each pair of auxiliary mask and box prompts, which can solve the requirements of extra prompts, is associated with class label predictions by the sum of the auxiliary classifier token and the learnable global classifier tokens in the mask decoder of SAM to solve the predictions of semantic labels. Meanwhile, we design a 3D depth-convolution adapter for image embeddings and a 3D depth-MLP adapter for prompt embeddings. We inject one of them into each transformer block in the image encoder and mask decoder to enable pre-trained 2D SAM models to extract 3D information and adapt to 3D medical images. Our method achieves state-of-the-art performance on AMOS2022, 90.52% Dice, which improved by 2.7% compared to nnUNet. Our method surpasses nnUNet by 1.7% on ACDC and 1.0% on Synapse datasets.
BibTeX
@misc{xie2024masksam,
  author = {Bin Xie and Hao Tang and Bin Duan and Dawen Cai and Yan Yan},
  title = {{MaskSAM: Towards Auto-prompt SAM with Mask Classification for Medical Image Segmentation}},
  year = {2024},
  eprint = {2403.14103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.14103},
}
Figure