Introduction
Compositional image retrieval (CIR) is a sophisticated task in computer vision where a system must identify a target image based on a reference image and a modification text. For instance, a user might provide an image of a black dress and the instruction "make it shorter and red." While large-scale models have made strides in this area, they typically require massive datasets of image-text-image triplets. PromptCLIP addresses the challenge of "few-shot" scenarios, where only a handful of training examples (e.g., 2 to 16) are available.
The Challenge of Data Scarcity
Existing state-of-the-art architectures, such as TIRG or FiLM, are notoriously data-hungry. In few-shot settings, these models often fail to converge or suffer from severe overfitting. Collecting the necessary triplets for every possible domain is expensive and time-consuming, creating a bottleneck for practical applications.
Methodology: The PromptCLIP Framework
PromptCLIP leverages the power of the pretrained CLIP (Contrastive Language-Image Pre-training) model through a parameter-efficient approach known as prompt tuning. The framework consists of several key innovations:
1. Visual Prompt Tuning
To avoid catastrophic forgetting and maintain the robustness of the pretrained model, the CLIP encoders are frozen. The reference image is converted into a trainable visual prompt by passing its embedding through a small, learnable linear layer. This projects the visual information into the input space of the CLIP text encoder.
2. Composed Query Sequence
By treating the visual prompt as a prefix to the modification text embeddings, the model allows the CLIP text encoder to process the combined sequence as a single multimodal query. This enables the model to utilize CLIP’s internal knowledge of cross-modal relationships to interpret how the text should modify the image.
3. Self-Supervised Auxiliary Task
A critical component of PromptCLIP is the introduction of an identity-preserving auxiliary task. The model is trained to retrieve the reference image itself when the modification text is empty (e.g., "The same as it is"). This acts as an autoencoding constraint, ensuring the visual prompt accurately captures the essence of the reference image before any modifications are applied.
Experimental Results
The authors evaluated PromptCLIP on three benchmarks: FashionIQ (clothing), CelebA (faces), and Birds to Words (species descriptions).
- CelebA (16-shot): PromptCLIP achieved a 43.5% Recall@5, significantly outperforming the best baseline's 28.5%.
- FashionIQ: In this notoriously difficult domain, traditional baselines often produced near-zero results in few-shot settings. PromptCLIP maintained a robust 21.2% Recall@10 at 16-shot.
- Ablation Studies: Removing the self-supervised auxiliary loss ($L_{aux}$) led to a sharp decline in performance, confirming its role in grounding the visual prompts.
Conclusion and Future Directions
PromptCLIP demonstrates that large-scale pretrained models can be adapted for complex compositional tasks with minimal data. By treating images as "visual words" and employing clever self-supervised constraints, the model bridges the gap between general-purpose pretraining and niche retrieval tasks. Future work may explore more complex mapping layers or multi-token visual prompts to capture even finer visual nuances.
Summary updated