Self-Prompt Mechanism for Few-Shot Image Recognition

Mingchen Song, Huiqiang Wang, Guoqiang ZhongAAAI 2024 · DOI

Abstract

Few-shot image recognition aims to learn new categories from very few examples. While prompt learning using text descriptions has shown promise, it often suffers from a "modality gap" between text and visual data. This paper introduces the Self-Prompt Mechanism (SPM), a novel approach that utilizes an image's own deep-layer features as prompts. By fine-tuning less than 2% of the model's parameters, SPM bridges the modality gap and establishes new performance benchmarks across standard and cross-domain few-shot learning datasets.

The Challenge: Beyond Text-Based Prompts

Traditional few-shot learning (FSL) methods have recently leaned on prompt learning, inspired by vision-language models like CLIP. These methods use text prompts (e.g., "A photo of a {bird}") to provide semantic hints to the visual encoder. However, the authors identify three major flaws in this approach:

  1. The Modality Gap: Text and images are fundamentally different. Aligning them perfectly is difficult and often leads to performance-degrading misalignments.
  2. Manual Overhead: Generating high-quality text prompts often requires human effort or computationally expensive language models.
  3. Visual Inaccuracy: Simple class names often fail to capture the rich visual diversity within a category.
Comparison between text semantic prompt and the proposed self-prompt mechanism.

The Self-Prompt Mechanism (SPM)

To overcome these limitations, the authors propose the Self-Prompt Mechanism (SPM), which treats the model’s own deep-layer features as internal "prompts." Built on a Vision Transformer (ViT) backbone, the architecture consists of three main stages:

1. Self-Prompt Generation

Instead of looking at external text, the system extracts semantic information from the features generated in the deep layers of the image encoder. It performs a feature selection process across both spatial (location) and channel (content) dimensions to identify the most relevant information.

2. Self-Prompt Injection

These extracted features are fed back into the last three layers of the Transformer. Specifically, they are added to the "Query" vectors in the self-attention mechanism. This effectively tells the model: "Based on what you've seen in the deep layers, here is what you should be looking for in the input."

Detailed architecture of the Self-Prompt Mechanism (SPM) showing the generation and injection of self-prompts within Transformer blocks.

3. Parameter-Efficient Tuning (PET)

During the adaptation phase, the backbone of the model remains frozen. Only the tiny amount of parameters related to self-prompt generation and layer biases are updated. This accounts for less than 2% of the total parameters, preventing the model from forgetting its prior knowledge while allowing it to specialize for new tasks.

Experimental Results

The researchers evaluated SPM on standard benchmarks and challenging cross-domain few-shot learning (CDFSL) tasks.

  • Standard Benchmarks: On Mini-ImageNet, SPM achieved 93.7% (1-shot) and 98.3% (5-shot) accuracy, surpassing previous leaders like PMF and SP-CLIP. On CIFAR-FS, it reached 82.4% in the 1-shot setting.
  • Cross-Domain Robustness: SPM showed significant improvements on datasets like EuroSAT (satellite) and CropDiseases (agriculture), with average performance gains of 3.65% and 2.33% respectively. This proves the mechanism is robust even when the target data differs significantly from the training domain.

Key Insights and Future Directions

SPM is inspired by human metacognition—the ability to adjust strategies based on internal feedback. By staying entirely within the visual domain, SPM avoids the modality gap inherent in text-based prompting. While the method introduces a slight increase in computational cost during the forward pass, its efficiency and accuracy make it a powerful tool for few-shot recognition.

Future work may focus on automating the selection of layers for prompt injection and exploring how this mechanism scales to even larger Vision Transformer architectures.

Summary updated

BibTeX
@inproceedings{song2024self,
  author = {Mingchen Song and Huiqiang Wang and Guoqiang Zhong},
  title = {{Self-Prompt Mechanism for Few-Shot Image Recognition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i5.28297},
}
Figure