Abstract
Prompt learning has become a popular approach for adapting large vision-language models, such as CLIP, to downstream tasks. Typically, prompt learning relies on a fixed prompt token or an input-conditional token to fit a small amount of data under full supervision. While this paradigm can generalize to a certain range of unseen classes, it may struggle when domain gap increases, such as in fine-grained classification and satellite image segmentation. To address this limitation, we propose Retrieval-enhanced Prompt learning (RePrompt), which introduces retrieval mechanisms to cache the knowledge representations from downstream tasks. We first construct a retrieval database from training examples, or from external examples when available. We then integrate this retrieval-enhanced mechanism into various stages of a simple prompt learning baseline. By referencing similar samples in the training set, the enhanced model is better able to adapt to new tasks with few samples. Our extensive experiments over 15 vision datasets, including 11 downstream tasks with few-shot setting and 4 domain generalization benchmarks, demonstrate that RePrompt achieves considerably improved performance. Our proposed approach provides a promising solution to the challenges faced by prompt learning when domain gap increases.
Introduction
Large-scale vision-language models like CLIP have revolutionized zero-shot recognition, but adapting them to specific "few-shot" downstream tasks (where only 1–16 examples per class are available) remains a significant challenge. Current adaptation methods generally fall into two categories: prompt tuning, which learns static "soft" tokens, and adapters, which use a cache of training features as an auxiliary classifier.
However, these approaches often fail to fully exploit the available training data. Static prompts do not adapt to specific input images, and simple adapters do not refine the underlying features extracted by the encoder. RePrompt (Retrieval-enhanced Visual Prompt learning) bridges this gap by making the prompting process dynamic and retrieval-aware. By "consulting" a database of training samples during inference, RePrompt generates instance-specific visual prompts that significantly improve performance in fine-grained classification and high-domain-gap scenarios.
Methods
RePrompt introduces a multi-stage retrieval mechanism that integrates knowledge from the training set directly into the model's architecture. The process consists of four main components:
- Retrieval Database: The model extracts features from the few-shot training set using a frozen CLIP image encoder, storing them as a key-value cache where image features are keys and labels are values.
- Retrieval-enhanced Visual Prompt Learner (REConv): For a new query image, the model retrieves the $k$-nearest neighbors from the cache. These neighbors, along with the query, are processed by a novel "REConv" block. This block uses 1x1 and 3x3 convolutions to fuse retrieved information into dynamic visual prompts inserted into the early layers of the Vision Transformer (ViT).
- Retrieval-based Adapter: Following the visual encoder, a $k$-NN classifier blends the standard CLIP prediction with a prediction based on feature similarity to the cached samples.
- Retrieval-guided Training: During fine-tuning, a reweighted cross-entropy loss uses $k$-NN confidence to focus the model's learning on "hard" examples that the retrieval mechanism finds ambiguous.
Experimental Results
The authors conducted an extensive evaluation across 15 datasets, including 11 image datasets (e.g., ImageNet, EuroSAT, DTD), 3 video datasets, and a multi-view dataset.
RePrompt consistently outperformed existing methods such as CoOP, Maple, and Tip-Adapter-F across almost all benchmarks. Key findings include:
- Significant Gains: RePrompt showed a +5.87% accuracy improvement on the Describable Textures Dataset (DTD) and a massive +9.04% on Stanford Cars compared to baseline visual prompt tuning.
- Efficiency: The model is highly efficient, requiring only 20 training epochs (vs. 200 for CoOP) and adding minimal inference latency (approx. 40ms).
- Robustness: In domain generalization tests (e.g., training on ImageNet and testing on ImageNet-Sketch), RePrompt maintained superior performance, indicating that the retrieval mechanism improves feature robustness rather than just memorizing the training set.
Conclusion
RePrompt demonstrates that dynamic, retrieval-enhanced prompting is a powerful strategy for few-shot adaptation of vision-language models. By leveraging local training data to inform visual prompts on-the-fly, the model achieves state-of-the-art results across a wide variety of vision tasks. Future work will investigate extending this framework to dense prediction tasks like semantic segmentation and object detection, where local retrieval could help identify specific object parts or textures.
Summary updated