Introduction
Vision-Language Models (VLMs) like CLIP have revolutionized zero-shot and few-shot learning. Adapting these models to downstream tasks is typically achieved through prompt learning, where a small set of "soft" tokens is tuned while keeping the backbone frozen. However, real-world scenarios often involve multiple related tasks, such as classifying different types of textures or satellite imagery.
Current adaptation strategies face a fundamental trade-off:
- Task-Individual Learning: Tuning a unique prompt for each task often leads to overfitting, particularly in low-data regimes, as it fails to leverage shared knowledge across tasks.
- Task-Shared Learning: Using a single global prompt for all tasks can cause "negative transfer," where conflicting information from diverse tasks (e.g., "dogs" vs. "tractors") degrades overall performance.
To address this, the authors propose Hierarchical Prompt Learning (HiPro), a framework that captures both broad category knowledge and fine-grained task details by organizing tasks into a hierarchical structure.
Methodology
HiPro operates through a multi-stage process designed to balance task-specific nuances with cross-task commonalities:
- Quantifying Inter-Task Affinity: The framework first measures the alignment of task gradients. High affinity is identified when two tasks suggest updates to a shared prompt in similar directions.
- Constructing the Hierarchical Task Tree: Using these affinity scores, agglomerative hierarchical clustering builds a tree. The root represents the union of all tasks, internal nodes represent semantic clusters (e.g., "all animals"), and leaf nodes represent individual tasks.
- Hierarchical Prompt Learning: A distinct prompt is learned for every node in the tree. Internal nodes capture "group-shared" features, while leaf nodes focus on "task-individual" details.
- Inference via Weight Fusion: During testing, classifier weights generated by all prompts along the path from the root to the specific task's leaf are averaged. This ensemble approach fuses general and specific knowledge.
Experimental Setup
The effectiveness of HiPro was evaluated across four major multi-task learning benchmarks:
- Office-Home: 4 domains (Art, Clipart, Product, Real-World) with 65 shared categories.
- DomainNet: A large-scale dataset featuring 6 domains and 345 categories.
- CIFAR-100: Structured into 20 tasks based on coarse super-categories.
- Large-Scale MTL Benchmark: A diverse collection of 10 datasets, including OxfordPets, EuroSAT (satellite), and DTD (textures).
Analysis and Insights
A key observation in this work is the regularization effect of hierarchical prompts. Task-individual prompts ($w_{ind}$) often achieve very low training loss but suffer from high test error due to overfitting. Conversely, shared prompts ($w_{shr}$) exhibit higher training loss but better generalization. HiPro's fusion mechanism ($w_{avg}$) acts as a semantic regularizer, finding a middle ground that optimizes for both training fit and test-time robustness.
Furthermore, the hierarchical structure allows the model to capture information at multiple semantic granularities. For instance, in CIFAR-100, the model can simultaneously learn features relevant to "animals," "medium mammals," and specific breeds like "dogs."
Results
HiPro consistently outperforms standard prompt learning (CoOp) and existing multi-task learning (MTL) baselines:
- DomainNet: HiPro achieved a ~6.7% accuracy improvement over CoOp in extremely low-data regimes (1% and 2% data).
- Office-Home: The method yielded a ~3% gain over multi-task CoOp variants.
- Few-Shot Robustness: Significant performance gains were observed across 1, 2, 4, 8, and 16-shot scenarios, highlighting its utility when training data is scarce.
- Efficiency: Since the VLM backbone remains frozen, training is compact. Additionally, there is no extra computational overhead during inference compared to other ensemble-based prompt methods.
Conclusion
While HiPro offers a robust middle ground for multi-task adaptation, it relies on a pre-constructed tree based on initial global prompt gradients. Future work may involve learning the tree structure dynamically or extending the hierarchical approach to multi-modal tasks beyond vision and language. Despite these areas for growth, HiPro demonstrates that multi-grained prompt learning is a powerful tool for bridging the gap between task-specific and global knowledge.
Summary updated