Adapting Frozen Promptable Segmenters to User-Defined Taxonomies via Few-Shot Proposal Learning

Khayrul Islam, Christopher Frederic MillerarXiv 2026 · arXiv

TL;DR

SNAP is a few-shot annotation pipeline that adapts frozen foundation models like SAM2.1 to specific project taxonomies by learning new class boundaries from just a few user labels. It significantly outperforms zero-shot foundation models on fine-grained tasks—such as distinguishing between eight look-alike cell lines—while refitting its project-specific classification head in seconds.

What Problem Does This Paper Address?

Foundation models like the Segment Anything Model (SAM) family have revolutionized image segmentation by making mask creation "cheap"—often requiring just a click or a box. However, the "costly half" of labeling remains: assigning the correct class.

Generic models are limited by their fixed training vocabulary. While they can easily separate a "person" from a "car," they struggle with the finer distinctions often required in specialized research, such as separating different truck models, identifying specific aircraft variants, or distinguishing between look-alike biological cell lines. Currently, researchers must either manually label every instance or perform expensive, time-consuming model fine-tuning. The authors identify that most annotation projects don't bring "new" objects, but rather require a new boundary between objects the foundation model has already seen.

Methods

The authors introduce SNAP, an annotation pipeline that keeps the heavy foundation model frozen and learns only the project-specific class boundaries.

Figure 2. Overview of SNAP. Gray blocks are frozen, blue blocks are fit to the labeled images L, and the dashed block is an optional proposal source. Every proposal passes through one frozen SAM2.1 decoder, and each newly labeled image refits SnapHead and the learned proposer.

Key components include:

  • SnapHead: A lightweight multinomial logistic regression head that classifies mask proposals. It is trained on-the-fly using the few labels provided by the annotator and refits in seconds.
  • Plug-in Proposals: SNAP uses SAM2’s standard Automatic Mask Generator (AMG) but adds a Learned Proposer. This second proposer is trained to find objects where the generic grid-based AMG fails, such as small, densely packed cells.
  • Rich Descriptors: Every mask proposal is described by a 512-dimensional vector that concatenates image embeddings and the decoder's own internal "object pointer," capturing both how the region looks and how the model segmented it.
  • The Router: When a more advanced model like SAM 3 is available, SNAP uses a "router" to decide class-by-class whether to trust the generic model's boundary or the newly learned SnapHead boundary based on their performance on the labeled data.

Datasets

The authors evaluated SNAP across a diverse set of 13 development projects and 5 held-out transfer datasets:

  • Natural & Aerial: COCO (vehicles, kitchen, animals), LVIS-rare, VOC-20, ADE20K-indoor, and iSAID (aerial buildings/vehicles).
  • Microscopy: LIVECell (including specific lines like A172 and SH-SY5Y) and BBBC038 nuclei.
  • Fine-Grained Ladders: Stanford Cars (body type vs. model), FGVC-Aircraft (manufacturer vs. variant), and ShipRSImageNet.
  • Transfer Set: Five datasets randomly drawn from the RF100-VL benchmark (e.g., stomata, floating waste, floods).

Key Insights and Ideas

  • Features Already Know the Way: A major "aha moment" is that frozen foundation features already encode the fine-grained boundaries a generic vocabulary misses. You don't need to retrain the whole model; you just need a better way to "read out" the existing features.
  • Vocabulary is a Fragile Interface: For specialized domains like microscopy, text prompts are notoriously unreliable. SAM 3 might find nuclei if prompted with "bright spots" but fail completely with the word "nucleus." SNAP bypasses this "prompt engineering" by learning directly from visual labels.
  • Granularity Matters: As the required classification becomes finer (e.g., moving from "car" to a specific "sedan model"), generic models quickly fall to chance performance. SNAP’s performance, however, scales steadily with the number of labels provided.
Figure 1. A generic segmenter has a fixed class boundary, but every project needs its own. Two annotation projects over the same scene, colored by class. SAM 3’s boundary (orange) is fixed by its training vocabulary and stays at the coarse person/car cut. A boundary learned from a project’s own labels (blue) moves to the finer sedan/truck/bus cut that project needs.

Key Results

  • Superior Fine-Grained Accuracy: On eight look-alike cell lines, SAM 3 achieved a Cohen’s $\kappa$ of only 0.038 (near chance). With 60 labeled images, SnapHead reached $\kappa = 0.754$.
  • Beating the Oracle: In end-to-end microscopy tasks with 20 labeled images, SNAP achieved a 0.484 F1 score, outperforming a SAM 3 "oracle" (0.441 F1) that had its prompts and thresholds perfectly tuned on the evaluation data.
  • Reduced Human Effort: SNAP significantly reduces the "drawn fraction"—the percentage of objects an annotator must draw by hand. On the LIVECell-A172 dataset, the fraction of manually drawn objects dropped from over 90% to roughly 50%.
  • Generalization: When run unchanged on five held-out datasets, SNAP beat the off-the-shelf SAM 3 on four of them.
Qualitative comparison on SH-SY5Y cells. SnapHead (right) correctly identifies nearly all cells that SAM 3 (middle) misses or misclassifies.

Limitations and Future Work

  • Localization Floor: The system is still limited by the underlying frozen encoder. For extremely small objects (e.g., small vehicles in high-res aerial tiles), the encoder's resolution can become a bottleneck for the proposer.
  • Class Competition: SnapHead uses a softmax classification, which assumes classes are mutually exclusive. Future work could explore multi-label setups for overlapping taxonomies.
  • Active Learning Integration: While SNAP currently uses random image selection for labeling, the authors suggest that integrating active learning (choosing the most "informative" images to label next) could drive annotation costs even lower.

Summary updated

Abstract

Promptable segmenters of the SAM family turn a click or a phrase into an instance mask, so the costly part of a label is now its class. However, each annotation project draws its own class boundary, often through objects these models already know: people from cars, sedans from trucks, one sedan model from another. A generic model's boundary is fixed by its training vocabulary. We present SNAP, an annotation pipeline that learns the project's boundary from the labels the annotator draws anyway. On one frozen SAM2.1 pass per image, we fit a per-project head, SnapHead, that classifies every proposal and refits in seconds after each labeled image. Where SAM 3 is available, the same labels decide class by class whether to keep its boundary or ours. Given the true masks of eight look-alike cell lines, SAM 3 assigns classes at Cohen's kappa = 0.038 and SnapHead at 0.754 (60 labeled images), and when only the class names change, SAM 3 falls to chance and SnapHead's score is unchanged. End to end, with 20 labeled images, SNAP beats a SAM 3 oracle tuned on the evaluation labels on five microscopy projects (0.484 against 0.441 F1), and run unchanged on five held-out datasets, it beats SAM 3 as shipped on four. Our results suggest that frozen foundation features already encode the class boundaries a generic vocabulary misses, and that a project's own labels are enough to read them out.
BibTeX
@misc{islam2026adapting,
  author = {Khayrul Islam and Christopher Frederic Miller},
  title = {{Adapting Frozen Promptable Segmenters to User-Defined Taxonomies via Few-Shot Proposal Learning}},
  year = {2026},
  eprint = {2611.10092},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10092},
}
Figure