Zero-Shot Segmentation via Frozen Vision Foundation Models

43 concepts36 papers1 proposal

Deep neural networks trained on broad visual tasks—such as image classification, image generation, or self-supervised contrastive learning—must implicitly learn the geometry of the visual world to succeed. Long before a model is explicitly trained to draw a pixel-perfect mask, its internal layers develop rich representations of edges, boundaries, and "objectness."

Rather than training new segmentation models from scratch for every novel domain, a growing body of research focuses on extracting and steering these pre-existing spatial priors. By leveraging frozen, pre-trained foundation models, it is possible to achieve precise object segmentation in a zero-shot or training-free manner across highly specialized domains.

The Intuition of Frozen Spatial Priors

Why can a model segment an object it has never been trained to see? The answer lies in the emergent geometric awareness of deep feature maps.

In early work on training-free localization, K J Joseph et al. demonstrated in MASON: A Model AgnoStic ObjectNess Framework that simply aggregating and summing the feature maps of a standard pre-trained classifier provides a robust localization signal. When combined with traditional, unsupervised graph-cut algorithms, these raw activation maps can be upgraded into zero-shot instance segmentation tools without any manual pixel-level annotations.

This intrinsic spatial awareness is not limited to discriminative classifiers; it also emerges naturally within generative and projection-based architectures:

Repurposing Foundation Models Across Disparate Domains

Modern vision foundation models, such as the Segment Anything Model (SAM) or DINOv2, formalize this intuition by mapping sparse prompts (such as points, bounding boxes, or text) to dense segmentation masks. Because their internal spatial priors are highly generalized, these models can be instantly repurposed for wildly disparate downstream tasks without updating their weights.

Environmental and Remote Sensing Tasks

In remote sensing, where manual annotation is exceptionally tedious, the zero-shot capabilities of foundation models provide immediate utility. For instance, Yougui Ren et al. deployed SAM directly to satellite imagery in Segment Anything Model (SAM) Assisted Remote Sensing Supervision for Mariculture to automatically isolate coastal mariculture structures.

Taking this a step further, Longchao Da et al. combined SAM’s zero-shot boundary extraction with simple chromatic analysis in Shaded Route Planning Using Active Segmentation and Identification of Satellite Images to map shaded pedestrian pathways from space, bypassing the need for specialized LiDAR data.

Bridging the Gap to Abstract Domains

Remarkably, these natural image priors are robust enough to generalize even to abstract, sparse domains. Mia Tang et al. demonstrated in Instance Segmentation of Scene Sketches Using Natural Image Priors that foundation models trained on natural photos inherently grasp boundaries within sparse line sketches. The authors revealed that the primary bottleneck in sketch segmentation is not generating the mask itself, but rather the upstream task of predicting object bounding boxes to prompt the model.

[Unlabeled Image] ──> [Frozen Foundation Model] ──> [Zero-Shot Masks] ──> [Dense Training Pairs]

Beyond direct inference, these zero-shot masks can bootstrap other learning systems. In SAM-DA: UAV Tracks Anything at Night with SAM-Powered Domain Adaptation, Liangliang Yao et al. introduced a "sample swelling" technique. By using SAM to automatically segment multiple potential entities within unlabeled nighttime images, the framework generates high-quality training pairs to provide dense supervision for downstream tracking models without human intervention.

Overcoming Data Scarcity in Medical Imaging

Medical imaging represents a critical frontier for zero-shot segmentation due to the extreme cost, scarcity, and expert variability associated with manual clinical annotations. Foundation models act as plug-and-play spatial decoders or pre-processors to bypass this bottleneck.

Spatial Guidance for Downstream Models

Instead of forcing a model to learn both "where to look" and "what to diagnose" simultaneously, zero-shot segmentation can decouple these challenges:

Generalizing to Unseen Pathologies and Modalities

For direct clinical segmentation, models can isolate specific organs from broader contexts without custom training. For example, Alireza Hekmat-Ardakani et al. leveraged SAM and DINOv2 to extract high-fidelity tongue regions from facial images in AI-driven tongue image analysis for diagnosing and predicting coronary artery disease.

When adapting to entirely new clinical tasks, the "One-Prompt" framework by Junde Wu et al. (One-Prompt to Segment All Medical Images) allows a model to generalize to unseen modalities and structures using only a single reference example as a prompt.

Furthermore, the authors of Towards a Comprehensive, Efficient and Promptable Anatomic Structure Segmentation Model Using 3D Whole-Body CT Scans, Heng Guo et al., demonstrated that models trained on broad anatomical datasets can successfully isolate un-modeled pathologies like tumors. They achieve this by leveraging their holistic understanding of normal, healthy anatomy alongside sparse user cues, outperforming specialized models fine-tuned on limited pathological data.

Steering Frozen Models at Inference Time

If the weights of a foundation model remain completely frozen, how can we guarantee high precision on complex, ambiguous, or highly specific targets? The solution lies in test-time adaptation and context-aware interaction encoding.

Test-Time Adaptation via Negative Mining

Instead of updating weights via backpropagation on a massive training set, a framework can adapt to a single image at inference time by evaluating its own internal consistency.

Jian Hu et al. developed this concept in INT: Instance-Specific Negative Mining for Task-Generic Promptable Segmentation. Given a generic task description (e.g., "find the polyp"), the system iteratively masks suspected object regions and measures how drastically a Vision-Language Model's confidence drops. By mining its own errors to refine instance-specific prompts at test time, the system generates precise masks for novel domains without any weight adjustments.

[Image + Task Description] ──> [Predict Mask] ──> [Inpaint/Mask Object] ──> [Measure VLM Confidence Drop] ──> [Refine Prompt]

For open-world scenarios driven by visual examples, Lv Tang et al. developed IPSeg (Towards Training-Free Open-World Segmentation via Image Prompt Foundation Models). This framework uses a single image prompt to guide frozen foundation models like DINOv2 and Stable Diffusion, bypassing task-specific fine-tuning entirely.

Context-Aware Interaction Encoding

When human guidance is available (such as clicks or bounding boxes), the mechanism used to encode those interactions determines how effectively the frozen model can adapt.

In MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning, Xiangde Luo et al. introduced Exponentialized Geodesic Distance (EGD) encodings. By forcing user click interaction maps to follow image intensity gradients rather than simple Euclidean distances, the interaction map itself carries structural context. This allows a frozen model to instantly adapt to the boundaries of unseen object types and modalities.

Similarly, Fang Yan et al. integrated click-based interaction tools with large foundation models in Artificial intelligence-based assessment of PD-L1 expression in diffuse large B cell lymphoma, achieving precise cell membrane delineation in complex pathological images.

These diverse applications demonstrate a unified paradigm: sparse human or textual guidance, when properly encoded, is highly effective at steering the robust, pre-existing spatial priors of frozen vision models to solve entirely new segmentation challenges.

Key Foundational Work

MASON: A Model AgnoStic ObjectNess Framework (2018) — contributed 1 concept elements to this direction.
Automatic Image Segmentation by Positioning a Seed (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_zero_shot_segmentation_00da6231,
  title = {zero-shot segmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-segmentation/rdr_00da62310cd447039f43fcc4f7811c75},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_zero_shot_segmentation_00da6231_v1,
  title = {zero-shot segmentation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/zero-shot-segmentation/rdr_00da62310cd447039f43fcc4f7811c75/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake