Self-Prompting Mechanisms for Vision Foundation Models

192 concepts134 papers3 proposals

Vision foundation models, most notably the Segment Anything Model (SAM) and its successors, have fundamentally changed the landscape of image segmentation. By framing segmentation as a promptable task, these models can isolate virtually any object given a spatial cue—such as a click, a bounding box, or a rough mask.

However, this paradigm introduces a critical bottleneck: it relies heavily on a human-in-the-loop to provide these prompts. While interactive segmentation is highly effective for asset creation or manual annotation, it fails in high-throughput, fully automated workflows like real-time medical imaging, large-scale remote sensing, or autonomous video analysis.

To bridge this gap, a major research direction focuses on self-prompting mechanisms (or automated prompt generation). Instead of eliminating the prompt encoder or completely retraining the foundation model, these approaches introduce lightweight, model-in-the-loop components that automatically generate optimal prompts directly from the input data. This preserves the rich, zero-shot delineation capabilities of the foundation model while removing the need for human intervention.


The Core Intuition: Decoupling Localization and Delineation

To understand why self-prompting works so well, we must look at image segmentation as two distinct sub-problems:

  1. Localization ("Where is it?"): Identifying the presence and rough spatial coordinates of a specific semantic class (e.g., finding a tumor in a CT scan or a car on a highway).
  2. Delineation ("Where are its exact boundaries?"): Tracing the precise, pixel-level contours of the identified object.

Foundation models like SAM are masters of delineation. Because they are trained on billions of masks, they possess an unmatched geometric intuition for object boundaries. However, they are class-agnostic; they do not inherently know which pixels represent a specific medical pathology or a target vehicle unless guided by a prompt.

Conversely, traditional, smaller deep learning architectures (like YOLO or lightweight UNets) excel at localization when trained on domain-specific data, but they often produce coarse, low-resolution boundaries when data is scarce.

Self-prompting architectures exploit this exact division of labor. By placing a small, task-specific "expert" model in front of a large foundation model, the expert handles the semantic localization and automatically passes its output as a spatial prompt to the foundation model, which executes the high-fidelity delineation.

[Input Image] ───► [Lightweight Expert Model] ───► [Automated Prompt] ───► [Frozen Foundation Model] ───► [Precise Mask]

Mechanism 1: Discrete Spatial Prompts (Detect-then-Segment)

The most intuitive way to automate prompting is to mimic human actions. If a human can guide a model by drawing a bounding box or clicking on an object, an object detection model can do the same.

In this paradigm, an auxiliary network processes the image first to generate explicit geometric coordinates. For instance, Mobina Mansoori et al. paired a YOLOv8 detector with SAM 2 for polyp segmentation in colonoscopies. The detector identifies the polyp and generates a bounding box, which serves as an immediate, automated prompt for the segmentation engine. Similarly, Yuanyuan Wang et al. demonstrated that extracting the center-point coordinates of bounding boxes can act as highly effective point prompts for mapping irregular road geometries.

To refine this process further, some frameworks employ a multi-stage or progressive prompting strategy. Xiuqi Zheng et al. introduced a "curriculum prompting" framework that moves systematically from coarse to fine-grained guidance. The system starts with an automated bounding box to generate an initial, rough mask, and then algorithmically derives precise edge coordinates from that intermediate mask to feed back into the foundation model for a final, highly polished boundary.

This strategy is also highly effective for automated data preparation. For example, Qiushi Guo combined object detection with saliency mapping to automatically prompt foundation models, creating high-quality object masks for context-aware data augmentation without manual pixel-level labeling. For nighttime surveillance, Guoxing Zhang built an automated workflow that translates initial object detections into precise oriented bounding boxes and masks, drastically reducing the manual effort required to annotate complex aerial datasets.


Mechanism 2: Learned Continuous Embeddings (Soft Prompting)

While discrete spatial prompts (like points and boxes) are highly interpretable, they suffer from a distinct flaw: they compress a wealth of visual and semantic information down to a few coordinate integers. If the detector makes a slight error in its bounding box, the downstream foundation model can easily be misled.

To solve this, a more sophisticated class of self-prompting mechanisms bypasses discrete geometry entirely. Instead, they train a lightweight network to translate the input image directly into continuous prompt embeddings—often referred to as "soft prompts"—that are injected straight into the foundation model's prompt encoder.

[Input Image] ───► [Lightweight Prompt Generator] ───► [Continuous Prompt Embedding] ───► [Frozen Prompt Encoder]

A pioneering example of this is found in the work of Tal Shaharabany et al., who introduced AutoSAM. Instead of calculating coordinates, they trained a lightweight, image-driven prompt generator that "overloads" SAM's frozen prompt encoder. The network learns to produce a surrogate embedding that contains rich, dense spatial and semantic cues tailored to specialized medical tasks.

Other approaches optimize these embeddings directly through advanced training formulations. Li Zhang et al. proposed a bi-level optimization framework (BLO-SAM) where learnable prompt embeddings are treated as hyperparameters. By optimizing these embeddings on a separate validation subset from the model weights, they achieved fully autonomous semantic segmentation while preventing overfitting in ultra-low data regimes.

In remote sensing, where objects like buildings or ships appear at arbitrary scales and orientations, Keyan Chen et al. developed RSPrompter. This architecture uses a Transformer-based encoder-decoder to learn "object queries" that automatically map into continuous prompt embeddings, allowing the model to perform category-aware instance segmentation without a single human click.


Mechanism 3: Prototypical and Few-Shot Self-Prompting

In scenarios where data is incredibly scarce—such as rare disease identification—training an auxiliary detector or an embedding generator from scratch is impossible. In these settings, researchers leverage prototypical networks to generate prompts based on feature similarity.

The intuition here is simple: if we have a single labeled example of a target object (a "support image"), we can extract its feature representation (the "prototype"). When a new, unlabeled image arrives, we can compute the pixel-wise similarity between the new image's features and the prototype. High-similarity regions naturally form a coarse mask, which can then be algorithmically converted into points or boxes to prompt the foundation model.

Lev Ayzenberg demonstrated this with ProtoSAM, a fully automated framework for one-shot medical image segmentation. By using a prototypical network to bridge the gap between class-agnostic foundation models and specific anatomical structures, ProtoSAM generates geometric prompts autonomously from a single exemplar.

Expanding on this concept to handle more complex variations, Zhonghao Yan et al. developed PGP-SAM. Their framework maps learned feature prototypes into both sparse point-like and dense mask-like representations simultaneously. This dual-prompting approach provides robust spatial guidance to the pre-trained segmentation decoder, enabling high-throughput clinical workflows that maintain the performance benefits of foundation models without requiring human interaction during inference.


Why This Paradigm Succeeds

Self-prompting mechanisms succeed because they respect the internal representations of pre-trained foundation models. Fine-tuning a model with hundreds of millions of parameters is computationally expensive and risks destroying the generalized features the model acquired during pre-training (a phenomenon known as catastrophic forgetting).

By leaving the massive foundation backbone frozen and focusing entirely on automating the input to the prompt encoder, self-prompting methods achieve the best of both worlds: the specialized accuracy of a domain expert and the robust, high-fidelity delineation of a foundation model.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 129 papers
  2. Version 1 · 2026-10-04 · 162 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_automated_prompt_generation_acc6afa0,
  title = {automated prompt generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-prompt-generation/rdr_acc6afa0e4b34a32968fb27d98d30b81},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_automated_prompt_generation_acc6afa0_v2,
  title = {automated prompt generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-prompt-generation/rdr_acc6afa0e4b34a32968fb27d98d30b81/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake