Semantic Prompt for Few-Shot Image Recognition

Liang WangCVPR 2023 · DOI · arXiv · DBLP

Abstract

Few-shot learning (FSL) remains a significant challenge in computer vision, as models often struggle to distinguish essential object features from spurious background noise when provided with limited examples. This paper introduces Semantic Prompt (SP), a novel framework that leverages textual descriptions to guide visual feature extraction. By integrating language-based "prompts" directly into a Vision Transformer (ViT) backbone via spatial and channel interactions, the model learns to prioritize relevant regions and features. Experimental results demonstrate that SP significantly enhances 1-shot recognition accuracy across major benchmarks, achieving state-of-the-art performance.

Introduction

The primary hurdle in Few-Shot Learning is the tendency of models to overfit to "spurious features"—background details or co-occurring objects that are not defining characteristics of the target class. For instance, a model shown a single image of a unicycle might mistakenly associate the surrounding brick wall or the rider with the concept of a "unicycle."

While prior work has attempted to use semantic information (such as word embeddings) to help, these methods typically apply semantic guidance only at the final classification stage. We argue that this is often too late; if the feature extractor has already focused on irrelevant noise, the classifier's ability to recover is limited. Inspired by the psychological concept of "cognitive penetrability"—where prior knowledge influences visual perception in real-time—we propose injecting semantic information directly into the feature extraction process.

Conceptual overview of Semantic Prompt-guided feature extraction, where a textual prompt guides the model to focus on the relevant object (e.g., a unicycle) and generate a precise attention map.

Methodology: The Semantic Prompt Framework

The Semantic Prompt (SP) architecture adaptively tunes a Vision Transformer (ViT) backbone using textual prompts. The system employs two primary mechanisms to bridge the gap between language and vision:

Spatial Interaction

In the spatial interaction module, class names are converted into "prompt vectors" and concatenated with the sequence of image patches. As these combined tokens pass through the Transformer's self-attention layers, the visual patches "attend" to the semantic prompt. This mechanism forces the model to spatially prioritize regions that align with the textual description, effectively filtering out background clutter.

Channel Interaction

Complementing spatial guidance, the channel interaction module modulates visual features on a per-channel basis. A small Multi-Layer Perceptron (MLP) processes the text embedding alongside global visual context to generate a modulating vector. This vector scales and shifts visual features, emphasizing specific "what" characteristics (e.g., "roundness" or "metallic texture") across the entire image.

The detailed architecture of the Semantic Prompt framework, illustrating the integration of a pre-trained text encoder with the spatial and channel interaction modules within a Transformer-based backbone.

Experimental Evaluation

We evaluated the SP framework on four standard FSL benchmarks: miniImageNet, tieredImageNet, CIFAR-FS, and FC100. The model utilizes a Visformer backbone—a hybrid of CNN and Transformer architectures—and undergoes a two-stage training process involving supervised pre-training followed by meta-learning fine-tuning.

Key Results

The Semantic Prompt approach achieved state-of-the-art results, particularly in the challenging 1-shot setting:

  • miniImageNet: 1-shot accuracy reached 72.31%, a substantial improvement over previous semantic-based methods like AM3 (65.30%).
  • tieredImageNet: Achieved 78.03% 1-shot accuracy.
  • Encoder Robustness: The method proved effective across various text encoders (CLIP, SBERT, GloVe), with CLIP yielding the best results due to its multi-modal pre-training.

Discussion and Insights

Our findings highlight several critical aspects of semantic-guided vision:

  1. Language as a Filter: Textual prompts act as a powerful filter during feature extraction, suppressing background noise and focusing the model on category-relevant details.
  2. The "Where" and the "What": The dual-mechanism approach effectively addresses both spatial localization (where the object is) and feature relevance (what the object looks like).
  3. Late-Stage Prompting: Ablation studies indicate that inserting prompts into the later stages of the network is most effective. While low-level features like edges can be shared across classes, high-level semantic features benefit most from targeted guidance.

Conclusion

The Semantic Prompt framework demonstrates that integrating language into the visual backbone is a highly effective strategy for few-shot recognition. By guiding the feature extractor with textual prompts, we can mitigate the impact of spurious features and improve model generalization. Future work may explore the use of more detailed textual descriptions or attributes to further refine the model's visual perception.

Summary updated

BibTeX
@inproceedings{wang2023semantic,
  author = {Liang Wang},
  title = {{Semantic Prompt for Few-Shot Image Recognition}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/CVPR52729.2023.02258},
}

Loading related content...

Figure