Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm

Jinrong Zhang, Penghui Wang, Chunxiao Liu, Wei Liu, Dian Jin, Qiong Zhang, Erli Meng, Zhengnan HuAAAI 2025 · OpenReview

TL;DR

This paper introduces a novel "Image Prompt Paradigm" for open-set visual perception tasks like object detection and segmentation. Instead of relying on text descriptions or interactive visual cues, it uses a few example images of a target category to guide the model, enabling single-stage, automated inference for specialized categories that are hard to describe.

What Problem Does This Paper Address?

Current methods for Open-Set Object Detection (OSOD) and Open-Set Segmentation (OSS) often struggle with specialized categories that are difficult to describe accurately with text. While visual prompts (like bounding boxes) can help, they typically require multi-round human interaction, making them impractical for fully automated pipelines. This paper aims to bridge this gap by proposing a new paradigm that leverages a small set of example images as prompts, allowing for efficient and automated open-set perception.

Methods

The core of the proposed solution is the Image Prompt Paradigm and the MI Grounding framework.

  • Image Prompt Paradigm: This paradigm uses a few cropped instances of target objects from training data as "image prompts." These prompts are used to guide the model in identifying and segmenting similar objects in new images, without needing textual descriptions or interactive annotation.
  • MI Grounding Framework: This framework comprises several key components:
  • Image Prompts Selection Encoder (IPS Encoder): This module is responsible for encoding, selecting, and integrating features from the input image prompts. It uses a pre-trained Vision Transformer (ViT) to extract semantic information and a Prompt Feature Selection Module (PFSM) based on self-attention. The PFSM is crucial for filtering out low-quality or outlier prompts, focusing on those with high semantic correlation.
  • Vision Encoder: A standard pre-trained ViT is used to extract multi-scale features from the input image.
  • Transformer Encoder with Deep Fusion: This component fuses the enhanced image prompt features with the multi-scale features from the input image using deformable cross-attention.
  • Region-Level Feature Alignment: Similar to text-based methods, this step aligns the prompt features with the predicted object embeddings.
  • Transformer Decoder: Finally, this decodes the fused features into bounding box labels and mask labels.

The framework is trained end-to-end using a combination of classification, localization (L1 and GIoU loss), and segmentation (cross-entropy) losses.

Datasets

The paper uses the following datasets:

  • Training: COCO, LVIS, Objects365.
  • Evaluation: COCO, LVIS, ODinW, ADE20K, SegInW.
  • New Benchmark: ADR50K (Automatic Defect Recognition dataset). This dataset is specifically constructed for X-ray defect inspection and contains over 50,000 images with annotations for three specialized defect categories: "sink," "shrinkage porosity," and "porosity." It highlights challenges in aligning category names with instances, confusion between similar category names, and distinguishing defects from background artifacts.

Key Insights and Ideas

  • Image Prompts as a Powerful Alternative: The paper demonstrates that a few example images can be as effective, if not more so, than textual descriptions for guiding open-set perception, especially for highly specialized or domain-specific categories.
  • The Importance of Prompt Quality: Not all image prompts are created equal. The paper highlights that low-quality or outlier prompts can degrade performance. The proposed PFSM effectively addresses this by using self-attention to select high-quality prompts.
  • Single-Stage, Non-Interactive Inference: The Image Prompt Paradigm, coupled with MI Grounding, achieves a significant advantage by enabling fully automated, single-stage inference, eliminating the need for human interaction during the detection or segmentation process.
  • Domain Generalization: The framework shows strong performance on out-of-domain datasets, suggesting good generalization capabilities.

Key Results

  • Competitive Performance: MI Grounding achieves competitive performance on standard OSOD and OSS benchmarks (COCO, LVIS, ADE20K, SegInW) compared to text and visual prompt methods.
  • Superiority on Specialized Dataset: On the newly constructed ADR50K dataset, MI Grounding significantly outperforms existing methods like Grounding DINO L and DINOV L, particularly for specialized defect categories. For instance, on ADR50K, MI Grounding achieves an AP of 50.2 for object detection and 50.4 for instance segmentation, outperforming text-based Grounding DINO L (43.4 APb, 43.1 APsink) and visual-based DINOV L (16.5 APb, 15.4 APsink).
  • Ablation Studies:
    • The PFSM module is shown to be more effective than simple mean pooling, FC, or CNN-based feature selection.
  • Increasing the image prompt update frequency during training improves performance.
  • An optimal image prompt quantity of 8 was found, balancing performance and computational cost.

Limitations and Future Work

  • Data Leakage Concern: While the paper states that test set data is not used for prompts, the reliance on training set crops for prompts could still be a concern in scenarios where training and testing distributions are very similar or if the training set is small.
  • Computational Cost: While an optimal prompt quantity of 8 was identified, using multiple image prompts inherently increases computational overhead compared to single-text prompts.
  • Future Work: The paper suggests that the Image Prompt Paradigm can be further explored for other vision tasks. The authors also imply that further research into more sophisticated prompt selection and integration strategies could yield even better results. The ADR50K dataset itself opens avenues for future research on specialized defect detection and segmentation.

Summary updated

Abstract

To break through the limitations of pre-training models on fixed categories, Open-Set Object Detection (OSOD) and Open-Set Segmentation (OSS) have attracted a surge of interest from researchers. Inspired by large language models, mainstream OSOD and OSS methods generally utilize text as a prompt, achieving remarkable performance. Following SAM paradigm, some researchers use visual prompts, such as points, boxes, and masks that cover detection or segmentation targets. Despite these two prompt paradigms exhibit excellent performance, they also reveal inherent limitations. On the one hand, it is difficult to accurately describe characteristics of specialized category using textual description. On the other hand, existing visual prompt paradigms heavily rely on multi-round human interaction, which hinders them being applied to fully automated pipeline. To address the above issues, we propose a novel prompt paradigm in OSOD and OSS, that is, Image Prompt Paradigm. This brand new prompt paradigm enables to detect or segment specialized categories without multi-round human intervention. To achieve this goal, the proposed image prompt paradigm uses just a few image instances as prompts, and we propose a novel framework named MI Grounding for this new paradigm. In this framework, high-quality image prompts are automatically encoded, selected and fused, achieving the single-stage and non-interactive inference. We conduct extensive experiments on public datasets, showing that MI Grounding achieves competitive performance on OSOD and OSS benchmarks compared to text prompt paradigm methods and visual prompt paradigm methods. Moreover, MI Grounding can greatly outperform existing method on our constructed specialized ADR50K dataset.
BibTeX
@inproceedings{zhang2025just,
  author = {Jinrong Zhang and Penghui Wang and Chunxiao Liu and Wei Liu and Dian Jin and Qiong Zhang and Erli Meng and Zhengnan Hu},
  title = {{Just a Few Glances: Open-Set Visual Perception with Image Prompt Paradigm}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=vnSPkkcA0P},
}
Figure