TL;DR
This paper introduces Aggregate-and-Adapted Prompt Embedding (AAPE), a novel prompt learning method that distills textual knowledge from LLMs to significantly improve CLIP's generalization on downstream vision-language tasks, especially in low-data and out-of-distribution scenarios. AAPE achieves state-of-the-art few-shot classification performance and competitive results on image captioning and VQA, all while eliminating LLM inference costs at test time.
What Problem Does This Paper Address?
Large pre-trained vision-language models like CLIP are powerful but often struggle with specialized domains (e.g., satellite imagery) or fine-grained classification (e.g., car models). This is because the visual concepts in these areas might be under-represented or entirely unseen during the model's massive pre-training. While prompt learning offers a parameter-efficient way to adapt these models, existing methods can be noisy or irrelevant, and some rely on costly LLM inference during testing. This paper aims to bridge this gap by leveraging rich textual knowledge from LLMs in a more effective and efficient manner for improved downstream generalization.
Methods
The core of the proposed method is the Aggregate-and-Adapt Prompt Embedding (AAPE). It involves two main stages:
- Prompt Aggregation: The process starts by generating natural language prompts for each image or class, either from human annotations or, more commonly, from LLMs like GPT-3. These raw prompts can be redundant or irrelevant. To address this, an "Input-Adapted Prompt Aggregator" is learned. This aggregator takes the embeddings of the reference prompts and, guided by CLIP's reward signal, distills them into a single, image-aligned "summary" prompt embedding. This step filters out noise and redundancy.
- Prompt Generation (AAPE): A "prompt generator" is then trained. This generator takes an input image and directly produces an "Aggregate-and-Adapted Prompt Embedding" (AAPE). This AAPE is trained using two losses:
- Distillation Loss ($L_{distill}$): Encourages the generated AAPE to stay close to the aggregated prompt summary obtained in the previous step. This transfers the distilled textual knowledge.
- Task Loss ($L_{task}$): Adapts the AAPE to the specific downstream task (e.g., classification, VQA).
Crucially, during inference, only the prompt generator is used, eliminating the need for LLM calls, making it highly efficient.
Datasets
The paper evaluates AAPE on a variety of datasets:
- Few-shot Image Classification: 11 datasets including ImageNet, Caltech101, OxfordPets, StanfordCars, Flowers102, Food101, FGVC-Aircraft, SUN397, DTD, EuroSAT, and UCF101. Domain generalization is also tested on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R.
- Image-to-Text Retrieval: COCO (for training) and Flickr30k (for evaluation).
- Image Captioning & VQA: COCO (for training) and NoCaps (for captioning evaluation), VQA2 (for VQA evaluation).
Key Insights and Ideas
- LLM Knowledge is Powerful, but Needs Adaptation: While LLMs provide rich textual priors, simply using them directly can be detrimental due to noise and irrelevance. AAPE's key innovation is learning to adapt this knowledge to the specific input image and task.
- Aggregation Filters Noise: The prompt aggregator effectively identifies and suppresses redundant or irrelevant prompts, creating a cleaner supervisory signal for the prompt generator.
- AAPE as a Universal Embedding: The learned AAPE can serve as a powerful, image-aligned text embedding that generalizes well across diverse vision-language tasks, not just classification.
- Efficiency Gains: By eliminating LLM inference at test time, AAPE offers significant computational advantages over previous LLM-based methods.
- Modality Gap Nuance: The paper suggests that reducing the "modality gap" (difference between image and text feature spaces) isn't always the sole driver of generalization. Multi-task learning, as employed by AAPE, can reshape the loss landscape to encourage more generalizable solutions, even if the raw modality gap isn't minimized.
Key Results
- Few-Shot Classification: AAPE achieves state-of-the-art performance on average across 11 datasets for base-to-new class generalization, with a new SOTA mean accuracy of 80.97%. It also shows strong performance in domain generalization settings.
- Image-to-Text Retrieval: AAPE demonstrates competitive zero-shot retrieval performance on Flickr30k, even when trained on COCO, highlighting its generalization capabilities.
- Image Captioning & VQA: AAPE shows consistent gains over baselines like LiMBER, and even outperforms the fully fine-tuned SOTA model MAGMA in terms of efficiency. It also shows improved captioning quality, especially in ambiguous cases.
- Efficiency: AAPE is about 2.8x faster for training and 1.2x faster for inference than MAGMA on Nvidia A100 GPUs. It also scales better with data and LLM size than alternatives.
Limitations and Future Work
- Prompt Aggregation Complexity: While the attention-based aggregator is efficient, aggregating into more than one prompt embedding to model text diversity is suggested but not fully explored due to increased complexity.
- LLM Inference Cost in Learning: While test-time LLM inference is eliminated, the initial LLM querying for prompt generation still incurs cost during the learning phase.
- Potential for Bias Amplification: The paper acknowledges that if the source text data (human captions or LLM prompts) contains biases, AAPE could inherit or amplify them, leading to unfair outcomes.
- Future Work: The authors suggest exploring learning diversified prompt generators, applying AAPE to other vision-language models beyond CLIP, and further investigating the dynamic relationship between multi-task learning, modality gap, and downstream generalization.
Summary updated