Few-Shot Composition Learning for Image Retrieval with Prompt Tuning

Junda Wu, Rui Wang, Handong Zhao, Ruiyi Zhang, Chaochao Lu, Shuai Li, Ricardo Henao

Introduction

Compositional image retrieval (CIR) is a sophisticated task in computer vision where a system must identify a target image based on a reference image and a modification text. For instance, a user might provide an image of a black dress and the instruction "make it shorter and red." While large-scale models have made strides in this area, they typically require massive datasets of image-text-image triplets. PromptCLIP addresses the challenge of "few-shot" scenarios, where only a handful of training examples (e.g., 2 to 16) are available.

The Challenge of Data Scarcity

Existing state-of-the-art architectures, such as TIRG or FiLM, are notoriously data-hungry. In few-shot settings, these models often fail to converge or suffer from severe overfitting. Collecting the necessary triplets for every possible domain is expensive and time-consuming, creating a bottleneck for practical applications.

Examples of compositional image retrieval tasks where a reference image and modification text are used to find a specific target image.

Methodology: The PromptCLIP Framework

PromptCLIP leverages the power of the pretrained CLIP (Contrastive Language-Image Pre-training) model through a parameter-efficient approach known as prompt tuning. The framework consists of several key innovations:

1. Visual Prompt Tuning

To avoid catastrophic forgetting and maintain the robustness of the pretrained model, the CLIP encoders are frozen. The reference image is converted into a trainable visual prompt by passing its embedding through a small, learnable linear layer. This projects the visual information into the input space of the CLIP text encoder.

2. Composed Query Sequence

By treating the visual prompt as a prefix to the modification text embeddings, the model allows the CLIP text encoder to process the combined sequence as a single multimodal query. This enables the model to utilize CLIP’s internal knowledge of cross-modal relationships to interpret how the text should modify the image.

3. Self-Supervised Auxiliary Task

A critical component of PromptCLIP is the introduction of an identity-preserving auxiliary task. The model is trained to retrieve the reference image itself when the modification text is empty (e.g., "The same as it is"). This acts as an autoencoding constraint, ensuring the visual prompt accurately captures the essence of the reference image before any modifications are applied.

Experimental Results

The authors evaluated PromptCLIP on three benchmarks: FashionIQ (clothing), CelebA (faces), and Birds to Words (species descriptions).

  • CelebA (16-shot): PromptCLIP achieved a 43.5% Recall@5, significantly outperforming the best baseline's 28.5%.
  • FashionIQ: In this notoriously difficult domain, traditional baselines often produced near-zero results in few-shot settings. PromptCLIP maintained a robust 21.2% Recall@10 at 16-shot.
  • Ablation Studies: Removing the self-supervised auxiliary loss ($L_{aux}$) led to a sharp decline in performance, confirming its role in grounding the visual prompts.

Conclusion and Future Directions

PromptCLIP demonstrates that large-scale pretrained models can be adapted for complex compositional tasks with minimal data. By treating images as "visual words" and employing clever self-supervised constraints, the model bridges the gap between general-purpose pretraining and niche retrieval tasks. Future work may explore more complex mapping layers or multi-token visual prompts to capture even finer visual nuances.

Summary updated

Abstract

We study the problem of composition learning for image retrieval, for which we learn to retrieve target images with search queries in the form of a composition of a reference image and a modification text that describes desired modifications of the image. Existing models of composition learning for image retrieval are generally built with large-scale datasets, demanding extensive training samples, ie, query-target pairs, as supervision, which restricts their application for the scenario of few-shot learning with only few query-target pairs available. Recently, prompt tuning with frozen pretrained language models has shown remarkable performance when the amount of training data is limited. Inspired by this, we propose a prompt tuning mechanism with the pretrained CLIP model for the task of few-shot composition learning for image retrieval. Specifically, we regard the representation of the reference image as a trainable visual prompt, prefixed to the embedding of the text sequence. One challenge is to efficiently train visual prompt with few-shot samples. To deal with this issue, we further propose a self-upervised auxiliary task via ensuring that the reference image can retrieve itself when no modification information is given from the text, which facilitates training for the visual prompt, while not requiring additional annotations for query-target pairs. Experiments on multiple benchmarks show that our proposed model can yield superior performance when trained with only few query-target pairs.
BibTeX
@inproceedings{wu2023few,
  author = {Junda Wu and Rui Wang and Handong Zhao and Ruiyi Zhang and Chaochao Lu and Shuai Li and Ricardo Henao},
  title = {{Few-Shot Composition Learning for Image Retrieval with Prompt Tuning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i4.25597},
}
Figure