TL;DR
This paper introduces DUDE, a novel prompt learning framework for vision-language models (VLMs) that leverages both general domain knowledge and specific class descriptions generated by LLMs. By employing Unbalanced Optimal Transport (UOT) to align visual features with these dual prompts, DUDE achieves superior performance in fine-grained classification tasks, outperforming existing state-of-the-art methods.
What Problem Does This Paper Address?
Large vision-language models (VLMs) like CLIP have shown remarkable capabilities in understanding images and text. A common way to adapt these models to new tasks without extensive fine-tuning is through "prompt learning," where carefully crafted text prompts guide the model's predictions. However, existing prompt learning methods often rely on a single, unified prompt for all classes. This can be insufficient for fine-grained classification tasks where subtle differences between categories are crucial. These methods can struggle to capture the unique attributes of each class, leading to misclassifications. Furthermore, many approaches use simple cosine similarity to match visual features with prompts, which might overlook more complex relationships and the fact that different parts of an image might correspond to different prompt elements.
Methods
DUDE tackles these limitations with a two-pronged approach:
- Dual Context Prompts: Instead of a single prompt, DUDE uses two types:
- Domain-Shared Prompts: These provide a general understanding of the task or domain, learned to be broadly applicable.
- Class-Specific Prompts: These are generated by Large Language Models (LLMs) like GPT, providing detailed, nuanced descriptions tailored to each specific class. This leverages the LLM's vast knowledge to capture unique class attributes.
- Unbalanced Optimal Transport (UOT): To measure the alignment between visual features and prompts, DUDE employs UOT. Unlike traditional Optimal Transport (OT) which enforces strict mass preservation, UOT allows for "unbalanced" distributions. This is beneficial because it can handle cases where some visual tokens might not have direct matches in the prompt set, or where there are noisy elements. UOT quantifies the "distance" between visual tokens and prompt embeddings, allowing for more robust and flexible matching. This is particularly useful when dealing with augmented images, as it can better align features from slightly altered images with the original prompts.
- Self-Attention Adapter: To manage the potentially large number of class-specific prompts, a shared self-attention module is used to compress and integrate these diverse prompts before alignment.
Datasets
The paper evaluates DUDE on five fine-grained datasets:
- Flowers102
- FGVCAircraft
- StanfordCars
- OxfordPets
- Food101
Key Insights and Ideas
- LLMs as Prompt Generators: Leveraging LLMs to generate class-specific prompts is a powerful way to imbue models with detailed, nuanced knowledge that is difficult to hand-craft. This significantly enhances the model's ability to differentiate between visually similar classes.
- UOT for Robust Alignment: Unbalanced Optimal Transport proves to be a more suitable metric than standard cosine similarity or balanced OT for aligning visual features with prompts, especially in the presence of noise, irrelevant information, or when dealing with augmented data. Its flexibility in handling mass distribution differences is key.
- Dual Prompts for Fine-Grained Tasks: The combination of general domain knowledge (shared prompts) and specific class details (LLM-generated prompts) is crucial for achieving high performance on challenging fine-grained classification tasks.
- Data Augmentation Synergy: UOT naturally complements data augmentation techniques, allowing for more effective learning from expanded datasets by better handling variations in visual features.
Key Results
DUDE demonstrates strong performance across various settings:
- Few-Shot Learning: In 4-shot per class settings on the five datasets, DUDE achieved the best performance in three out of five settings and an average accuracy of 76.84%. It significantly outperformed zero-shot CLIP and single shared-prompt methods like Coop, notably achieving a 10.65% margin on StanfordCars.
- Base-to-New Generalization: In a 16-shot setting for base-to-new class generalization, DUDE again showed superior performance in both base and new class settings, indicating its strong ability to generalize to unseen categories.
- Adapter-Based Settings: DUDE also outperformed previous adapter-based methods across different shot numbers and datasets, demonstrating its versatility.
Limitations and Future Work
- Computational Cost: While UOT is efficient, the generation of class-specific prompts from LLMs can add to the overall computational overhead.
- Future Work: The authors propose to:
- Test DUDE on various adapter-based learning scenarios to further validate its generalization capabilities.
- Extend the framework to vision-language models trained with autoregressive settings (e.g., LLAVA), which presents a unique challenge due to differences in their learned embedding space structures compared to contrastively trained models like CLIP.
Summary updated