All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning

Can Cui, Ruining Deng, Quan Liu, Tianyuan Yao, Shunxing Bao, Lucas W. Remedios, Yucheng Tang, Yuankai Huo

Abstract

The Segment Anything Model (SAM) is a recently proposed prompt-based segmentation model in a generic zero-shot segmentation approach. With the zero-shot segmentation capacity, SAM achieved impressive flexibility and precision on various segmentation tasks. However, the current pipeline requires manual prompts during the inference stage, which is still resource intensive for biomedical image segmentation. In this paper, instead of using prompts during the inference stage, we introduce a pipeline that utilizes the SAM, called all-in-SAM, through the entire AI development workflow (from annotation generation to model finetuning) without requiring manual prompts during the inference stage. Specifically, SAM is first employed to generate pixel-level annotations from weak prompts (e.g., points, bounding box). Then, the pixel-level annotations are used to finetune the SAM segmentation model rather than training from scratch. Our experimental results reveal two key findings: 1) the proposed pipeline surpasses the state-of-the-art (SOTA) methods in a nuclei segmentation task on the public Monuseg dataset, and 2) the utilization of weak and few annotations for SAM finetuning achieves competitive performance compared to using strong pixel-wise annotated data.

Introduction

The Segment Anything Model (SAM) has revolutionized computer vision with its "zero-shot" ability to segment objects based on prompts like points or boxes. However, it struggles with medical images—specifically pathology slides—because medical data was underrepresented in its training set. To obtain high-quality results on nuclei, standard SAM usage typically requires a manual bounding box for every single nucleus during the inference (testing) stage. In a single pathology slide containing thousands of nuclei, this is practically impossible for a human to perform. While finetuning SAM is an option, standard finetuning usually requires expensive, pixel-perfect manual annotations. The authors address this "annotation bottleneck" by creating a workflow that works with weak labels and requires zero manual effort during actual model deployment.

Methodology: The All-in-SAM Pipeline

The authors propose a two-stage pipeline called All-in-SAM, which integrates SAM into both the annotation and the learning phases of model development.

  1. SAM-Empowered Annotation: Instead of asking experts to draw complex pixel-wise outlines, they provide "weak" annotations (simple bounding boxes). These boxes are fed into the pretrained SAM to generate high-quality "pseudo-labels" (approximate masks). This leverages SAM’s inherent ability to find edges when given a hint.
  2. Prompt-based Finetuning: These pseudo-labels are used to finetune the SAM model using a "SAM-adapter" approach. This involves keeping the heavy Vision Transformer (ViT) backbone mostly frozen while training lightweight "adapter" layers to learn the specific textures and features of nuclei.
The All-in-SAM architecture for training and inference.

Unlike standard SAM usage, the final finetuned model is designed to segment nuclei without any prompts during inference. The model learns to "prompt itself" based on the domain knowledge gained during finetuning, effectively bridging the gap between foundation models and specialized medical tasks.

Experimental Setup and Datasets

The study utilizes the MICCAI 2018 Monuseg dataset, a benchmark for multi-organ nuclei segmentation. The training set consists of 30 images (1000x1000 pixels), and the test set includes 14 images. To test the model's robustness in data-scarce environments, the authors also experimented with "few-shot" scenarios, training on as little as 0.5% of the available data.

Key Results

The All-in-SAM pipeline demonstrates significant improvements over existing methods in both accuracy and efficiency.

  • State-of-the-Art Performance: On the Monuseg dataset, All-in-SAM achieved a Dice score of 82.54% and an IoU of 69.74%, surpassing other SOTA methods like LViT and nnUNet.
  • Superiority with Weak Labels: When trained only on weak annotations (boxes), the proposed method achieved a 82.46% Dice score, which is nearly identical to the performance achieved with perfect pixel-wise labels.
  • Low-Data Excellence: In extreme cases where only 0.5% of data was used, All-in-SAM maintained a Dice score of 78.73%, significantly outperforming nnUNet (75.00%) in the same category.
Qualitative comparison of All-in-SAM vs. nnUNet across different data regimes.

Visual results show that All-in-SAM produces cleaner, more distinct boundaries for individual nuclei compared to the often "aggressive" or merged segmentations produced by traditional models like nnUNet.

Conclusion and Future Work

The All-in-SAM pipeline effectively "upgrades" weak annotations into high-quality masks, reducing human labor by orders of magnitude without sacrificing performance. While highly successful, the authors noted that in some low-data scenarios, the model had lower recall than nnUNet, occasionally missing smaller nuclei. Additionally, the use of the large ViT-H backbone introduces higher computational costs. Future work will explore whether this "weak-to-strong" finetuning generalizes to other complex medical structures like glands or vessels across different imaging modalities.

Summary updated

Abstract

The Segment Anything Model (SAM) is a recently proposed prompt-based segmentation model in a generic zero-shot segmentation approach. With the zero-shot segmentation capacity, SAM achieved impressive flexibility and precision on various segmentation tasks. However, the current pipeline requires manual prompts during the inference stage, which is still resource intensive for biomedical image segmentation. In this paper, instead of using prompts during the inference stage, we introduce a pipeline that utilizes the SAM, called all-in-SAM, through the entire AI development workflow (from annotation generation to model finetuning) without requiring manual prompts during the inference stage. Specifically, SAM is first employed to generate pixel-level annotations from weak prompts (e.g., points, bounding box). Then, the pixel-level annotations are used to finetune the SAM segmentation model rather than training from scratch. Our experimental results reveal two key findings: 1) the proposed pipeline surpasses the state-of-the-art (SOTA) methods in a nuclei segmentation task on the public Monuseg dataset, and 2) the utilization of weak and few annotations for SAM finetuning achieves competitive performance compared to using strong pixel-wise annotated data.
BibTeX
@article{cui2024all,
  author = {Can Cui and Ruining Deng and Quan Liu and Tianyuan Yao and Shunxing Bao and Lucas W. Remedios and Yucheng Tang and Yuankai Huo},
  title = {{All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning}},
  journal = {Journal of Physics Conference Series},
  year = {2024},
  doi = {10.1088/1742-6596/2722/1/012012},
}
Figure