Spatial Prompt Sensitivity in Interactive Segmentation

91 concepts67 papers1 proposal

Interactive segmentation models, most notably the Segment Anything Model (SAM), have shifted the paradigm of computer vision by allowing users to define targets through spatial prompts like points, bounding boxes, and scribbles. However, a growing body of research reveals that these models are often remarkably fragile. A shift of just a few pixels in a click or a slightly loose bounding box can lead to a total collapse in segmentation quality. This phenomenon, known as spatial prompt sensitivity, represents a significant hurdle for deploying foundation models in high-stakes environments like medical imaging and industrial inspection.

The Fragility of the Click

The core intuition behind promptable segmentation is that a spatial cue acts as a guide for the model's attention. Ideally, a model should be robust enough to understand that a click near the center of a liver or a box around a car implies the whole object, regardless of minor placement errors.

In practice, however, the performance landscape is often "non-smooth." As Moskalenko et al. demonstrate through the TETRIS benchmark, state-of-the-art models exhibit extreme sensitivity to minor perturbations. A click moved by a tiny margin can cause the model to switch from segmenting a target to segmenting the background or an adjacent structure. This sensitivity is particularly acute in scientific and medical imaging. Price found that in microscopy, where low contrast and "halo" effects are common, even points placed slightly outside the target can cause significant performance degradation.

The Impact of Human Variability

This sensitivity creates a "domain gap" between the idealized, robotic prompts used during model training and the messy, imprecise inputs provided by human experts. Huang et al. observed this in medical workflows, noting that while SAM is a powerful tool for human-in-the-loop annotation, its instability makes it unreliable for fully automated tasks where prompts might be generated by noisy secondary algorithms.

The problem is compounded by human subjectivity. As noted by Zhang et al., different users perceive the "center" or "boundary" of complex objects differently. If a model requires pixel-perfect accuracy to function, as Luo et al. argue is the case for many deep learning frameworks, it becomes difficult to use for anyone but the most meticulous operators. This lack of robustness to "casual" prompts—inputs that are roughly correct but not precise—limits the utility of foundation models in rapid clinical or industrial settings.

From Flaw to Uncertainty

While sensitivity is generally viewed as a weakness, some researchers have repurposed it as a diagnostic tool. Li et al. introduced A-SAM, which treats SAM’s sensitivity to minor prompt shifts as a way to model uncertainty. By slightly perturbing a prompt and observing the variance in the resulting masks, the authors can identify "ambiguous" regions where even the model is unsure of the boundary. This is especially valuable for medical lesions where expert disagreement is common; the model's sensitivity effectively maps the range of plausible interpretations.

Engineering for Robustness

To move beyond this fragility, recent work focuses on "taming" or calibrating the model's response to spatial noise. Several distinct strategies have emerged:

  • Deformable Calibration: Fan et al. proposed Stable-SAM, which adds a lightweight deformable sampling module to prevent "attention drift." This allows the model to maintain high-quality segmentation even when provided with noisy or off-center bounding boxes.
  • Noise-Tolerant Architectures: New frameworks like NTClick specifically target intricate, thin geometries, allowing users to be less precise with their clicks without sacrificing mask quality.
  • Domain-Aware Adaptation: For low-resource settings, Chen et al. developed the FEMed framework, which uses selective adaptation to handle "coarse" prompts, ensuring that medical segmentation doesn't collapse when the user provides a loose bounding box.
  • Automated Refinement: Rather than forcing the user to be perfect, Lin et al. introduced SAMRefiner, which uses the model itself to clean up noisy initial masks, effectively acting as an error-correction mechanism for imprecise inputs.

By shifting the focus from absolute accuracy to stability—the consistency of the output across a range of varying-quality prompts—the field is moving toward interactive systems that are not just powerful, but reliable in the hands of human users.

Key Foundational Work

Geodesic Thin Plate Splines for Image Segmentation (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_prompt_sensitivity_spatial_prompt_sensit_b346273a,
  title = {prompt sensitivity (spatial prompt sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prompt-sensitivity-spatial-prompt-sensitivity/rdr_b346273a986447d287d878aab4ba2dd6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_prompt_sensitivity_spatial_prompt_sensit_b346273a_v1,
  title = {prompt sensitivity (spatial prompt sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/prompt-sensitivity-spatial-prompt-sensitivity/rdr_b346273a986447d287d878aab4ba2dd6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake