All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning

Can Cui, Ruining Deng, Quan Liu, Tianyuan Yao, Shunxing Bao, Lucas W. Remedios, Bennett A. Landman, Yucheng Tang, Yuankai Huo

TL;DR

This paper introduces "All-in-SAM," a novel pipeline that leverages the Segment Anything Model (SAM) for efficient nuclei segmentation in biomedical images. By using SAM to generate pixel-level annotations from weak prompts (like bounding boxes) and then finetuning SAM with these generated labels, the method significantly reduces annotation burden and achieves state-of-the-art performance without requiring manual prompts during inference.

What Problem Does This Paper Address?

Biomedical image segmentation, particularly for tasks like nuclei segmentation, is crucial for research and clinical applications. However, it's notoriously labor-intensive and time-consuming due to the need for precise pixel-level annotations. While powerful foundation models like SAM offer impressive zero-shot segmentation capabilities, their current workflow often requires manual prompts during inference, which remains a bottleneck for large-scale biomedical applications. Furthermore, SAM's performance can be suboptimal in specialized domains like medical imaging, especially when prompt data is limited. This paper aims to bridge this gap by developing a method that automates annotation generation and enables efficient finetuning of SAM using weak and limited annotations, thereby reducing the overall cost and effort of developing high-performing segmentation models for biomedical tasks.

Methods

The proposed "All-in-SAM" pipeline consists of two main stages:

  1. SAM-Empowered Annotation: Instead of manual annotation, the pretrained SAM model is used to generate approximate pixel-level masks. This is achieved by providing weak annotations, specifically bounding boxes around nuclei, as input to SAM. These generated masks serve as "pseudo ground truth" for the next stage.
  2. SAM-Finetuning: The generated pixel-level annotations are then used to finetune the SAM model. This finetuning process adapts the generic SAM to the specific task of nuclei segmentation. Crucially, this finetuning does not require manual prompts during the inference stage, making the final model more user-friendly and efficient. The authors explore finetuning strategies that leverage weak annotations (bounding boxes) and limited data.

Datasets

The primary dataset used in this work is the MICCAI 2018 Monuseg dataset. This dataset consists of 30 training images and 14 testing images, each with dimensions of 1000x1000 pixels, and includes corresponding nuclei masks. The authors used the same data split as a recent study, dividing the 30 training images into a 24-image training set and a 6-image validation set.

Key Insights and Ideas

  • Automated Annotation Generation: The core idea of using SAM itself to generate high-quality pseudo-labels from weak annotations is a significant departure from traditional manual annotation. This drastically reduces the annotation bottleneck.
  • Weak Annotation for Finetuning: The paper demonstrates that finetuning SAM with weak annotations (bounding boxes) and limited data can yield performance comparable to, or even exceeding, methods that rely on complete pixel-wise annotations.
  • Prompt-Free Inference: A key outcome is a finetuned SAM model that no longer requires manual prompts during inference, making it directly applicable for segmentation tasks after the initial training phase.
  • Efficiency in Biomedical Imaging: The proposed pipeline directly addresses the resource-intensive nature of biomedical image segmentation, offering a more practical and scalable solution.

Key Results

The experimental results highlight the effectiveness of the All-in-SAM pipeline:

  • State-of-the-Art Performance: The proposed pipeline surpasses state-of-the-art methods on the Monuseg dataset for nuclei segmentation, particularly when using weak and limited annotations.
  • Competitive Performance with Limited Data: Even with as little as 4% or 0.5% of the training data, the All-in-SAM pipeline achieves competitive segmentation performance, demonstrating its label-efficiency. For instance, with 4% weak annotation, the proposed method achieved a Dice score of 81.34%, which is comparable to or better than many other methods using more data.
  • Robustness to Annotation Quality: The method shows resilience to annotation quality, performing well with weak labels where other methods might struggle. For example, when using weak labels, the proposed method consistently outperformed nnUNet across various metrics.
  • Qualitative Improvements: Figure 3 visually demonstrates the qualitative results, showing that the proposed method, even with limited data, produces segmentation masks that are visually comparable to or better than those from nnUNet.

Limitations and Future Work

  • Limitations:
    • While the paper demonstrates strong performance with weak annotations, the initial generation of these weak annotations (bounding boxes) still requires some manual effort, though significantly less than full pixel-wise annotation.
  • The paper notes that in some extreme cases with very limited and weakly annotated data, the proposed method exhibited a lower Recall value compared to nnUNet, although it maintained higher precision.
  • Future Work:
    • The authors aim to further enhance the application of SAM in nuclei segmentation by addressing annotation burden and cost issues through label-efficient finetuning and the utilization of weak annotations.
  • The pipeline's success suggests potential for adaptation to other biomedical segmentation tasks beyond nuclei.

Summary updated

Abstract

The Segment Anything Model (SAM) is a recently proposed prompt-based segmentation model in a generic zero-shot segmentation approach. With the zero-shot segmentation capacity, SAM achieved impressive flexibility and precision on various segmentation tasks. However, the current pipeline requires manual prompts during the inference stage, which is still resource intensive for biomedical image segmentation. In this paper, instead of using prompts during the inference stage, we introduce a pipeline that utilizes the SAM, called all-in-SAM, through the entire AI development workflow (from annotation generation to model finetuning) without requiring manual prompts during the inference stage. Specifically, SAM is first employed to generate pixel-level annotations from weak prompts (e.g., points, bounding box). Then, the pixel-level annotations are used to finetune the SAM segmentation model rather than training from scratch. Our experimental results reveal two key findings: 1) the proposed pipeline surpasses the state-of-the-art methods in a nuclei segmentation task on the public Monuseg dataset, and 2) the utilization of weak and few annotations for SAM finetuning achieves competitive performance compared to using strong pixelwise annotated data.
BibTeX
@article{cui2024all,
  author = {Can Cui and Ruining Deng and Quan Liu and Tianyuan Yao and Shunxing Bao and Lucas W. Remedios and Bennett A. Landman and Yucheng Tang and Yuankai Huo},
  title = {{All-in-SAM: from Weak Annotation to Pixel-wise Nuclei Segmentation with Prompt-based Finetuning}},
  journal = {Journal of Physics Conference Series},
  year = {2024},
  doi = {10.1088/1742-6596/2722/1/012012},
}
Figure