TL;DR
This paper introduces MDPR, a novel framework that enhances vision-language models (VLMs) for tasks with imbalanced datasets (long-tailed distributions). By leveraging LLMs to create a rich, multi-dimensional knowledge base of class semantics and employing a dynamic routing mechanism, MDPR significantly improves recognition accuracy, especially for under-represented "tail" classes, without substantial computational overhead.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) like CLIP have shown remarkable capabilities in understanding images and text. However, when fine-tuned on datasets with imbalanced class distributions (long-tailed data), they tend to perform poorly on classes with few examples (tail classes) while excelling on classes with many examples (head classes). This bias stems from both the pre-training data of VLMs and the imbalanced nature of downstream tasks. While Large Language Models (LLMs) have been used to inject semantic information, they often overlook the inherent biases within VLMs themselves, leading to cumulative errors. This paper aims to address this challenge by developing a method that effectively mitigates class imbalance during VLM fine-tuning.
Methods
The proposed Multi-dimensional Dynamic Prompt Routing (MDPR) framework consists of two main stages:
- Offline Multi-dimensional Prompt Construction:
- MDPR leverages LLMs to build a comprehensive knowledge base for each class. This knowledge base goes beyond simple class names and captures semantics across five dimensions:
- General Appearance (GA): Typical visual features (e.g., color, shape).
- Fine-grained Appearance (FA): Specific local details for distinguishing similar objects.
- Functionality (FT): The primary purpose of the object.
- Contextual Information (CI): Common environments or associated objects.
- Differential Features (DF): Unique characteristics that differentiate a class from its most confused counterpart (identified using a confusion matrix from CLIP's zero-shot predictions).
- This multi-faceted approach aims to dilute the uni-dimensional biases present in standard VLM representations.
- Online Dynamic Prompt Routing:
- During fine-tuning, an image-attentive semantic extraction module uses the input image's visual features to dynamically select and aggregate relevant semantic information from the knowledge base.
- A Class-specific Multi-Head Attention (C-MHA) module computes attention weights, forming an attentive semantic feature for each class. This routing mechanism is designed to be relevance-based, increasing inter-class margins for tail classes and reducing intra-class variance for head classes.
- The final prediction is a fusion of the base VLM's logits and the dynamically routed semantic logits, controlled by a hyperparameter $\beta$.
- Regularization losses are introduced to stabilize the de-biasing process and improve the quality of learned representations.
MDPR is designed as a plug-and-play enhancement that can be integrated with existing VLM fine-tuning methods.
Datasets
The authors evaluated MDPR on three widely adopted long-tailed image recognition benchmarks:
- CIFAR-100-LT: A long-tailed version of CIFAR-100.
- ImageNet-LT: A long-tailed version of ImageNet.
- Places-LT: A long-tailed version of Places.
Detailed statistics for these datasets, including the number of classes, training, and testing items, are provided.
Key Insights and Ideas
- Beyond Simple Prompts: The core idea that a single class name or a simple prompt is insufficient for robust VLM fine-tuning under imbalance is crucial. MDPR's multi-dimensional approach captures richer semantic nuances.
- LLMs as Semantic Augmenters, Not Just Generators: Instead of just generating more data, LLMs are used to build a structured knowledge base that directly informs the VLM's understanding of class semantics.
- Dynamic Routing is Key: The ability to dynamically select relevant semantic information based on the input image is critical. This allows the model to adapt its understanding and focus on discriminative features for tail classes.
- Differential Features are Highly Important: The ablation study highlights that the "Differential Features" dimension, which explicitly contrasts a class with its confusable counterparts, has the largest impact on accuracy, underscoring the importance of inter-class discrimination.
- Universality and Efficiency: MDPR is shown to be effective across different base VLM frameworks (CoOp, MaPLe) and backbones (ViT, ResNet), with only a modest increase in parameters and training time, making it a practical enhancement.
Key Results
- MDPR consistently achieves significant performance gains across all class groups (Many, Medium, Few) on CIFAR-100-LT, ImageNet-LT, and Places-LT.
- It demonstrates state-of-the-art (SOTA) or highly competitive results, particularly for Few-shot classes, which are the most challenging in long-tailed distributions.
- On Places-LT, MDPR (with MaPLe) boosted Few-shot accuracy by over 22% compared to the base MaPLe method.
- MDPR shows robustness across different backbones (ViT-B/16, ResNet-50, ResNet-101), consistently outperforming standard fine-tuning methods like CoOp, especially on ResNet backbones where CoOp struggles.
- Ablation studies confirm that each component of MDPR (semantic prompts and regularization) contributes to performance improvements, with the combination yielding the best results.
Limitations and Future Work
- Device Limitations: The effectiveness of MDPR has been primarily validated on CLIP ViT-B/16 and further verified on ResNet backbones. Further examination on larger-scale or different VLM architectures is needed.
- Reliance on Known Class Distributions: MDPR's prediction balancing still relies on known class distribution information from the training set. Future work could explore integrating methods like causal inference for better adaptability to open environments with unknown or dynamic class distributions.
- Offline Knowledge Base Construction: The current multi-dimensional semantic knowledge base is constructed offline. Future research could explore dynamic construction and updating mechanisms, and integrate real-world metadata or cross-validate with multiple LLMs to enhance reliability and scalability for incremental or open-set learning scenarios.
Summary updated