BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation

Prithwijit Chowdhury, Mohit Prabhushankar, Ghassan AlRegib

Abstract

The Segment Anything Model (SAM) has revolutionized interactive segmentation through spatial prompting. While existing work primarily focuses on automating prompts in various settings, real-world annotation workflows involve iterative refinement where annotators observe model outputs and strategically place prompts to resolve ambiguities. Current pipelines typically rely on the annotator's visual assessment of the predicted mask quality. We postulate that a principled approach for automated interactive prompting is to use a model-derived criterion to identify the most informative region for the next prompt. In this work, we establish active prompting: a spatial active learning approach where locations within images constitute an unlabeled pool and prompts serve as queries to prioritize information-rich regions, increasing the utility of each interaction. We further present BALD-SAM: a principled framework adapting Bayesian Active Learning by Disagreement (BALD) to spatial prompt selection by quantifying epistemic uncertainty. To do so, we freeze the entire model and apply Bayesian uncertainty modeling only to a small learned prediction head, making intractable uncertainty estimation practical for large multi-million parameter foundation models. Across 16 datasets spanning natural, medical, underwater, and seismic domains, BALD-SAM demonstrates strong cross-domain performance, ranking first or second on 14 of 16 benchmarks. We validate these gains through a comprehensive ablation suite covering 3 SAM backbones and 35 Laplace posterior configurations, amounting to 38 distinct ablation settings. Beyond strong average performance, BALD-SAM surpasses human prompting and, in several categories, even oracle prompting, while consistently outperforming one-shot baselines in final segmentation quality, particularly on thin and structurally complex objects.

Introduction

The Segment Anything Model (SAM) has fundamentally changed the landscape of image segmentation by enabling zero-shot performance through user-provided prompts such as clicks or bounding boxes. However, in practical applications, segmentation is rarely a one-shot process. Instead, it is an iterative "dialogue" where a human provides a prompt, evaluates the model's output, and then manually determines where to place the next prompt to correct errors.

This human-centric approach is inherently subjective and often inefficient. Human annotators may place prompts in redundant areas or fail to identify the specific regions that would provide the most significant "information gain" for the model's internal logic. To address this, we propose a shift toward model-driven interactive prompting, where the system itself identifies the most informative regions to resolve its own ambiguities.

Methodology: Active Prompting with BALD-SAM

We introduce BALD-SAM, a framework that reframes interactive segmentation as a spatial active learning problem. By treating every pixel in an image as a potential query in an unlabeled pool, the model can strategically select the next prompt location to maximize information utility.

The BALD-SAM interactive segmentation loop, illustrating how inclusion and exclusion prompts are used to refine the SAM output mask through iterative comparison.

The Lightweight Bayesian Head

SAM is a massive foundation model with over 600 million parameters, making traditional Bayesian inference computationally prohibitive. To make uncertainty estimation practical, we employ a "frozen backbone" strategy. We freeze the entire SAM architecture and append a lightweight, trainable prediction head consisting of approximately 35,000 parameters.

Quantifying Disagreement

We apply a Laplace approximation to estimate the posterior distribution of this small prediction head. This allows us to sample multiple plausible versions of the model efficiently. By running these samples, the model generates an ensemble of potential masks. BALD-SAM then calculates a "disagreement map" based on the Bayesian Active Learning by Disagreement (BALD) objective. The next prompt is automatically placed in the region where the ensemble members disagree most (high mutual information), specifically targeting epistemic uncertainty—the uncertainty stemming from the model's lack of knowledge rather than image noise.

Experimental Setup

To test the robustness and cross-domain utility of BALD-SAM, we evaluated the method across 16 datasets in four distinct domains:

  • Natural Images: Nine categories from MS COCO, including complex objects like "Stop signs" and "Ties."
  • Medical Imaging: Breast ultrasound, colonoscopy polyps, and skin lesions.
  • Underwater Imagery: The NDD20 dataset, featuring dolphins in varied environments.
  • Seismic Interpretation: The Netherlands F3 dataset, focusing on salt domes and chalk groups.

We compared our automated prompts against the PointPrompt dataset, which contains sequences of actual human-provided prompts, to establish a realistic performance baseline.

Key Results and Insights

Superior Efficiency and Accuracy

BALD-SAM demonstrated exceptional performance, ranking first or second in normalized $\Delta$ IoU (the improvement in mask quality per prompt) on 14 of the 16 benchmarks. It proved particularly effective for thin and structurally complex objects; for example, on the "Tie" category, it achieved an IoU of 0.845 compared to 0.649 for standard one-shot geometric baselines.

Surpassing Human Intuition

The framework consistently matched or exceeded the efficiency of human annotators. In specialized domains like medical and underwater imaging, where human visual intuition may be less reliable, BALD-SAM provided a more stable and predictable annotation experience with lower variance than human-provided prompts. Remarkably, in certain natural image categories like "Dog" or "Stop sign," BALD-SAM even outperformed "Oracle" prompting (prompts derived from ground-truth knowledge), suggesting that the model's internal uncertainty is a superior guide for its own refinement.

Epistemic vs. Aleatoric Uncertainty

A critical finding was the importance of targeting epistemic uncertainty. While standard entropy-based methods often get distracted by naturally blurry boundaries or noise (aleatoric uncertainty), BALD-SAM’s focus on model disagreement ensures that prompts are placed where they will most effectively expand the model's understanding of the object structure.

Conclusion and Future Work

BALD-SAM provides a practical blueprint for applying active learning to large-scale foundation models. By isolating Bayesian modeling to a lightweight head, we achieve high-utility interactive segmentation without the need for expensive re-training.

While the method is highly effective, limitations remain in domains where the underlying SAM backbone lacks representative features, such as seismic imaging. Future research will explore extending this active framework to other prompt modalities, such as bounding boxes or scribbles, and investigating data-efficient ways to fine-tune the backbone itself using these active queries.

Summary updated

Abstract

The Segment Anything Model (SAM) has revolutionized interactive segmentation through spatial prompting. While existing work primarily focuses on automating prompts in various settings, real-world annotation workflows involve iterative refinement where annotators observe model outputs and strategically place prompts to resolve ambiguities. Current pipelines typically rely on the annotator's visual assessment of the predicted mask quality. We postulate that a principled approach for automated interactive prompting is to use a model-derived criterion to identify the most informative region for the next prompt. In this work, we establish active prompting: a spatial active learning approach where locations within images constitute an unlabeled pool and prompts serve as queries to prioritize information-rich regions, increasing the utility of each interaction. We further present BALD-SAM: a principled framework adapting Bayesian Active Learning by Disagreement (BALD) to spatial prompt selection by quantifying epistemic uncertainty. To do so, we freeze the entire model and apply Bayesian uncertainty modeling only to a small learned prediction head, making intractable uncertainty estimation practical for large multi-million parameter foundation models. Across 16 datasets spanning natural, medical, underwater, and seismic domains, BALD-SAM demonstrates strong cross-domain performance, ranking first or second on 14 of 16 benchmarks. We validate these gains through a comprehensive ablation suite covering 3 SAM backbones and 35 Laplace posterior configurations, amounting to 38 distinct ablation settings. Beyond strong average performance, BALD-SAM surpasses human prompting and, in several categories, even oracle prompting, while consistently outperforming one-shot baselines in final segmentation quality, particularly on thin and structurally complex objects.
BibTeX
@misc{chowdhury2026bald,
  author = {Prithwijit Chowdhury and Mohit Prabhushankar and Ghassan AlRegib},
  title = {{BALD-SAM: Disagreement-based Active Prompting in Interactive Segmentation}},
  year = {2026},
  eprint = {2603.10828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.10828},
}
Figure