Abstract
Large vision-language models like CLIP demonstrate remarkable zero-shot capabilities but often struggle with specialized domains and "tail class" concepts under-represented in their training data. While Large Language Models (LLMs) contain the encyclopedic knowledge necessary to bridge these gaps, leveraging them typically incurs significant computational costs or introduces noise during inference. This paper presents AAPE (Aggregate-and-Adapted Prompt Embedding), a framework that distills rich textual descriptions from LLMs into a lightweight, image-conditioned prompt generator. By aggregating noisy LLM outputs into image-aligned summaries and adapting them to downstream tasks, AAPE achieves state-of-the-art performance in few-shot classification and vision-language tasks without requiring LLM access at test time.
1. Introduction
Vision-language models (VLMs) like CLIP have revolutionized computer vision by enabling open-vocabulary recognition. However, their performance often degrades when faced with fine-grained categories (e.g., specific car models) or specialized domains (e.g., satellite imagery). Standard adaptation techniques, such as prompt learning, often fail to leverage the deep semantic knowledge stored in Large Language Models. Conversely, methods that do use LLMs often require querying them for every image at inference, which is computationally prohibitive and sensitive to LLM hallucinations.
We introduce AAPE, a method designed to bake LLM wisdom directly into CLIP’s adaptation process. AAPE operates by distilling diverse textual descriptions into a lightweight prompt generator, ensuring that the resulting model is both semantically rich and computationally efficient.
2. Methodology
The AAPE framework is structured around two primary phases: Aggregate and Adapt.
2.1 The Prompt Aggregator
To capture the "encyclopedic" knowledge of a category, the framework first queries an LLM (or utilizes human-curated captions) to generate multiple descriptive prompts. Because LLMs can be prone to hallucinations or provide descriptions irrelevant to a specific visual instance, AAPE employs an attention-based aggregator. This module utilizes a "CLIP reward" mechanism to weight prompts based on their relevance to the image, producing a condensed, image-aligned summary embedding ($p^a$).
2.2 The Prompt Generator
To eliminate the need for an LLM during inference, AAPE trains a lightweight neural network—the Prompt Generator ($h$)—to predict the summary embedding directly from visual features. This allows the model to generate task-specific, semantically informed prompts using only the image encoder.
2.3 Joint Optimization
The generator is optimized using a dual-loss objective:
- Distillation Loss: Minimizes the distance between the generated prompt and the LLM-derived summary embedding.
- Task Loss: Ensures the generated prompt optimizes performance on the specific downstream task, such as classification or Visual Question Answering (VQA).
3. Experimental Evaluation
3.1 Datasets and Benchmarks
The authors evaluated AAPE across a comprehensive suite of benchmarks:
- Classification: 11 datasets including ImageNet, StanfordCars, OxfordPets, EuroSAT, and DTD.
- Robustness: Out-of-Distribution (OOD) benchmarks such as ImageNet-V2, Sketch, A, and R.
- Vision-Language Tasks: COCO and Flickr30k for image-to-text retrieval and VQA.
3.2 Key Results
- State-of-the-Art Generalization: AAPE achieved a mean accuracy of 80.97% across 11 datasets in base-to-new generalization settings, outperforming previous methods like OGEN.
- Data Efficiency: AAPE demonstrates remarkable few-shot capabilities; specifically, AAPE using only 1-shot (one labeled example) frequently outperforms standard prompt learning methods that utilize 16-shots.
- Inference Efficiency: Compared to heavy-duty vision-language models like MAGMA, AAPE is approximately 1.2x faster at inference and 2.8x faster to train, as the LLM is discarded after the distillation phase.
4. Discussion and Insights
4.1 Noise Filtering and Disambiguation
AAPE’s attention mechanism effectively filters LLM "noise." For instance, if an LLM describes a car's headlights but the image only shows the rear, the aggregator de-prioritizes the irrelevant textual cue. This capability is vital for disambiguation, helping CLIP distinguish between similar categories (e.g., "altar" vs. "church") by focusing on specific textual cues that correspond to subtle visual details.
4.2 The Modality Gap
Analysis of the "modality gap" (the distance between image and text embeddings) suggests that AAPE does not necessarily function by closing this gap. Instead, it reshapes the loss landscape, allowing the model to find solutions that generalize better to unseen data and out-of-distribution samples.
5. Conclusion and Future Work
AAPE provides a scalable and efficient bridge between the semantic depth of LLMs and the visual grounding of CLIP. While the current implementation aggregates knowledge into a single embedding, future iterations may explore predicting sets of embeddings to better capture complex scenes. Nonetheless, AAPE represents a significant advancement in making foundation models adaptable to the "long tail" of real-world data without sacrificing production-level efficiency.
Summary updated