Benchmarking Human and Automated Prompting in the Segment Anything Model

Jorge Quesada, Zoe Fowler, Mohammad Alotaibi, Mohit Prabhushankar, Ghassan AlRegib

Abstract

The Segment Anything Model (SAM) has established a new paradigm in image segmentation through its zero-shot capabilities driven by prompt engineering. However, the qualitative difference between human-generated prompts and automated heuristics remains poorly understood, particularly in specialized domains. This paper benchmarks human versus automated prompting strategies across diverse datasets, revealing a significant 29% performance gap. We demonstrate that while human intuition currently outperforms standard automated methods, finetuning SAM’s prompt encoder can significantly bridge this divide. Furthermore, we identify interpretable geometric features that drive successful segmentation, providing a roadmap for more effective automated prompting agents.

Introduction

As a foundation model for computer vision, the Segment Anything Model (SAM) relies on prompts—such as points or bounding boxes—to identify and isolate objects within an image. While much research has focused on the model's underlying architecture, the "quality" of the prompt itself is often overlooked. In practical applications, especially in "out-of-distribution" (OOD) domains like medical imaging or seismic analysis, the difference between a well-placed point and a random one can be the difference between a successful segmentation and a total failure.

This study quantifies the "human-machine gap" by comparing human-selected points against six common automated strategies. By evaluating these methods across 16 diverse categories, we aim to identify the specific factors that make a prompt effective and determine if SAM can be optimized to better interpret automated inputs.

Methodology

The researchers utilized the PointPrompt dataset, a curated collection of 6,000 images spanning natural images (COCO, NDD20), medical imaging (Chest-X, Kvasir-SEG, ISIC), and seismic data (F3 Facies). The study was structured around three core objectives:

  1. Performance Benchmarking: Human prompts were compared against six automated strategies: Random, K-Medoids (cluster-based), Shi-Tomasi (corner detection), Entropy, Maximum Distance, and Saliency-based sampling. The evaluation considered both inclusion points (target area) and exclusion points (background/ignore areas).
  2. Prompt Encoder Finetuning: To assess whether SAM could adapt to specific automated heuristics, the researchers froze the heavy image encoder and mask decoder, finetuning only the lightweight prompt encoder.
  3. Feature Decoding: The team analyzed 26 interpretable features—such as "prompt coverage" and "inclusion max spread"—to determine their mathematical correlation with segmentation success (Intersection over Union, or IoU).

Key Insights and Results

The Human Advantage

The study found a substantial performance disparity: automated strategies resulted in an average 29% decline in IoU compared to human prompters. This suggests that humans possess an intuitive understanding of an object's "essence" that simple geometric or saliency-based algorithms lack.

The Power of Inclusion

A critical finding was the disproportionate impact of inclusion points. Replacing automated inclusion points with human-selected ones yielded a 36.3% performance boost. Conversely, swapping exclusion points had negligible or even slightly negative effects, indicating that the primary driver of SAM's success is the accuracy of the positive prompt rather than the refinement of the negative one.

Bridging the Gap via Finetuning

Finetuning the prompt encoder proved highly effective. By training the model to better interpret specific automated strategies, the researchers improved performance by 22% to 68% over the base SAM model. Among the automated baselines, K-Medoids emerged as the most robust strategy, consistently outperforming other heuristics across various domains.

Domain Sensitivity and Predictability

The relationship between prompt features and performance is not universal. While no single feature predicts success across all datasets, the study found that in specialized OOD domains (medical and seismic), the relationship is much stronger (R² > 0.5). In seismic data, the gap between humans and machines was at its widest, suggesting that human experts use highly specific domain knowledge that current algorithms fail to replicate.

Discussion and Future Work

While finetuning the prompt encoder is effective, it remains computationally expensive because the model must perform full inference to calculate loss. However, the identification of "interpretable features" offers a path forward.

Instead of relying on generic automated strategies, future research can focus on developing domain-specific prompting agents. These agents could be programmed to optimize for specific geometric features—like "max spread" for natural objects or "boundary margin" for medical anomalies—tailoring the prompting strategy to the specific requirements of the task at hand. This move toward "principled" automated prompting could eventually eliminate the need for human intervention in complex segmentation pipelines.

Summary updated

BibTeX
@inproceedings{quesada2024benchmarking,
  author = {Jorge Quesada and Zoe Fowler and Mohammad Alotaibi and Mohit Prabhushankar and Ghassan AlRegib},
  title = {{Benchmarking Human and Automated Prompting in the Segment Anything Model}},
  booktitle = {IEEE Big Data},
  year = {2024},
  doi = {10.1109/bigdata62323.2024.10825731},
}
Figure