Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model

Duy M. H. Nguyen, An T. Le, Trung Q. Nguyen, Nghiem T. Diep, Tai Nguyen, Duy Duong-Tran, Jan Peters, Li Shen, Mathias Niepert, Daniel Sonntag

TL;DR

This paper introduces DUDE, a novel prompt learning framework for vision-language models (VLMs) that leverages both general domain knowledge and specific class descriptions generated by LLMs. By employing Unbalanced Optimal Transport (UOT) to align visual features with these dual prompts, DUDE achieves superior performance in fine-grained classification tasks, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Large vision-language models (VLMs) like CLIP have shown remarkable capabilities in understanding images and text. A common way to adapt these models to new tasks without extensive fine-tuning is through "prompt learning," where carefully crafted text prompts guide the model's predictions. However, existing prompt learning methods often rely on a single, unified prompt for all classes. This can be insufficient for fine-grained classification tasks where subtle differences between categories are crucial. These methods can struggle to capture the unique attributes of each class, leading to misclassifications. Furthermore, many approaches use simple cosine similarity to match visual features with prompts, which might overlook more complex relationships and the fact that different parts of an image might correspond to different prompt elements.

Methods

DUDE tackles these limitations with a two-pronged approach:

  • Dual Context Prompts: Instead of a single prompt, DUDE uses two types:
  • Domain-Shared Prompts: These provide a general understanding of the task or domain, learned to be broadly applicable.
  • Class-Specific Prompts: These are generated by Large Language Models (LLMs) like GPT, providing detailed, nuanced descriptions tailored to each specific class. This leverages the LLM's vast knowledge to capture unique class attributes.
  • Unbalanced Optimal Transport (UOT): To measure the alignment between visual features and prompts, DUDE employs UOT. Unlike traditional Optimal Transport (OT) which enforces strict mass preservation, UOT allows for "unbalanced" distributions. This is beneficial because it can handle cases where some visual tokens might not have direct matches in the prompt set, or where there are noisy elements. UOT quantifies the "distance" between visual tokens and prompt embeddings, allowing for more robust and flexible matching. This is particularly useful when dealing with augmented images, as it can better align features from slightly altered images with the original prompts.
  • Self-Attention Adapter: To manage the potentially large number of class-specific prompts, a shared self-attention module is used to compress and integrate these diverse prompts before alignment.

Datasets

The paper evaluates DUDE on five fine-grained datasets:

  • Flowers102
  • FGVCAircraft
  • StanfordCars
  • OxfordPets
  • Food101

Key Insights and Ideas

  • LLMs as Prompt Generators: Leveraging LLMs to generate class-specific prompts is a powerful way to imbue models with detailed, nuanced knowledge that is difficult to hand-craft. This significantly enhances the model's ability to differentiate between visually similar classes.
  • UOT for Robust Alignment: Unbalanced Optimal Transport proves to be a more suitable metric than standard cosine similarity or balanced OT for aligning visual features with prompts, especially in the presence of noise, irrelevant information, or when dealing with augmented data. Its flexibility in handling mass distribution differences is key.
  • Dual Prompts for Fine-Grained Tasks: The combination of general domain knowledge (shared prompts) and specific class details (LLM-generated prompts) is crucial for achieving high performance on challenging fine-grained classification tasks.
  • Data Augmentation Synergy: UOT naturally complements data augmentation techniques, allowing for more effective learning from expanded datasets by better handling variations in visual features.

Key Results

DUDE demonstrates strong performance across various settings:

  • Few-Shot Learning: In 4-shot per class settings on the five datasets, DUDE achieved the best performance in three out of five settings and an average accuracy of 76.84%. It significantly outperformed zero-shot CLIP and single shared-prompt methods like Coop, notably achieving a 10.65% margin on StanfordCars.
  • Base-to-New Generalization: In a 16-shot setting for base-to-new class generalization, DUDE again showed superior performance in both base and new class settings, indicating its strong ability to generalize to unseen categories.
  • Adapter-Based Settings: DUDE also outperformed previous adapter-based methods across different shot numbers and datasets, demonstrating its versatility.

Limitations and Future Work

  • Computational Cost: While UOT is efficient, the generation of class-specific prompts from LLMs can add to the overall computational overhead.
  • Future Work: The authors propose to:
  • Test DUDE on various adapter-based learning scenarios to further validate its generalization capabilities.
  • Extend the framework to vision-language models trained with autoregressive settings (e.g., LLAVA), which presents a unique challenge due to differences in their learned embedding space structures compared to contrastively trained models like CLIP.

Summary updated

Abstract

Prompt learning methods are gaining increasing attention due to their ability to customize large vision-language models to new domains using pre-trained contextual knowledge and minimal training data. However, existing works typically rely on optimizing unified prompt inputs, often struggling with fine-grained classification tasks due to insufficient discriminative attributes. To tackle this, we consider a new framework based on a dual context of both domain-shared and class-specific contexts, where the latter is generated by Large Language Models (LLMs) such as GPTs. Such dual prompt methods enhance the model's feature representation by joining implicit and explicit factors encoded in LLM knowledge. Moreover, we formulate the Unbalanced Optimal Transport (UOT) theory to quantify the relationships between constructed prompts and visual tokens. Through partial matching, UOT can properly align discrete sets of visual tokens and prompt embeddings under different mass distributions, which is particularly valuable for handling irrelevant or noisy elements, ensuring that the preservation of mass does not restrict transport solutions. Furthermore, UOT's characteristics integrate seamlessly with image augmentation, expanding the training sample pool while maintaining a reasonable distance between perturbed images and prompt inputs. Extensive experiments across few-shot classification and adapter settings substantiate the superiority of our model over current state-of-the-art baselines.
BibTeX
@inproceedings{nguyen2024dude,
  author = {Duy M. H. Nguyen and An T. Le and Trung Q. Nguyen and Nghiem T. Diep and Tai Nguyen and Duy Duong-Tran and Jan Peters and Li Shen and Mathias Niepert and Daniel Sonntag},
  title = {{Dude: Dual Distribution-Aware Context Prompt Learning For Large Vision-Language Model}},
  booktitle = {ACML Conference Track},
  year = {2024},
  url = {https://openreview.net/forum?id=SZ4LNs0zfE},
}
Figure