Abstract
Soft prompt learning methods are effective for adapting vision-language models (VLMs) to downstream tasks. Nevertheless, empirical evidence reveals a tendency of existing methods that they overfit seen classes and exhibit degraded performance on unseen classes. This limitation is due to the inherent bias in the training data towards the seen classes. To address this issue, we propose a novel soft prompt learning method, named Mixture-of-Prompts Distillation (MoPD), which can effectively transfer useful knowledge from hard prompts manually hand-crafted (a.k.a. teacher prompts) to the learnable soft prompt (a.k.a. student prompt), thereby enhancing the generalization ability of soft prompts on unseen classes. Moreover, the proposed MoPD method utilizes a gating network that learns to select hard prompts used for prompt distillation. Extensive experiments demonstrate that the proposed MoPD method outperforms state-of-the-art baselines especially on on unseen classes.
Introduction
Large-scale Vision-Language Models (VLMs) like CLIP are renowned for their zero-shot capabilities, enabling them to recognize objects through natural language descriptions known as "hard prompts" (e.g., "a photo of a [CLASS]"). To further refine these models for specific tasks, researchers developed Soft Prompt Learning (e.g., CoOp), which replaces hand-written prompts with learnable prompt vectors. While this significantly improves performance on classes seen during training (base classes), it often leads to overfitting, causing the model to lose its ability to generalize to new, unseen classes. The core challenge is maintaining the flexibility of learnable soft prompts while retaining the robust generalization of human-designed hard prompts.
Methodology
The proposed Mixture-of-Prompts Distillation (MoPD) framework treats prompt adaptation as a teacher-student problem.
The framework consists of three primary components:
- The Student: A learnable soft prompt vector similar to standard soft prompting methods.
- The Teachers: A pool of multiple hand-crafted hard prompts (e.g., "a type of texture" or "an image labeled as [CLASS]").
- The Gating Network: This lightweight network acts as a "prompt router." It analyzes the input image and dynamically decides which hard prompts from the pool are the most relevant "teachers" for that specific instance.
The model is optimized using a multi-part objective: a standard Cross-Entropy Loss ($\zeta_{CE}$) for classification, a Distillation Loss ($\zeta_{MPD}$) that aligns the student with the weighted mixture of teachers, and a Selection Loss ($\zeta_{MPS}$) that encourages the gating network to favor teachers that provide accurate guidance. Crucially, the gating network is only used during training; at inference, the model uses the optimized soft prompt alone, ensuring no added computational overhead.
Core Concepts
A key insight of MoPD is that multiple teachers are better than one. Different hard prompts capture distinct nuances of the VLM's pre-trained knowledge. By distilling from a mixture rather than a single prompt, the model gains a more comprehensive and robust representation.
Furthermore, instance-specificity plays a vital role. Not every image is best described by the same prompt template; an image with a complex background might benefit from different linguistic guidance than a simple studio shot. The gating network allows the model to dynamically choose the best guidance for each training sample, effectively filtering out "noisy" or irrelevant prompts for a given context.
Experimental Results
MoPD was evaluated across 11 diverse image classification datasets, including generic objects (ImageNet), fine-grained categories (OxfordPets, Food101), and specialized domains (EuroSAT, UCF101).
- Superior Generalization: In base-to-new generalization tasks, MoPD achieved the highest Harmonic Mean (H) score—a metric balancing base and new class accuracy—in 10 out of 11 datasets. Across all datasets, MoPD improved accuracy on "New" classes to 74.69%, compared to 68.00% for the baseline CoOp.
- Efficiency: While multi-modal methods like MaPLe perform well, they require millions of tunable parameters. MoPD achieves competitive or superior results with only ~8K parameters, making it significantly lighter to train.
- Domain Robustness: On ImageNet variants (V2, Sketch, A, and R), MoPD demonstrated strong performance, outperforming established baselines in handling style shifts.
Discussion and Conclusion
While MoPD shows state-of-the-art performance, it slightly underperformed on the OxfordPets dataset, likely due to a very small category shift from CLIP's original training data, which limited the utility of the hard prompts. Future work could explore using Large Language Models (LLMs) to automatically generate even more diverse teacher prompt pools or refining the gating architecture for even more precise prompt routing.
In summary, MoPD provides a robust and efficient solution for VLM adaptation, bridging the gap between task-specific learning and broad generalization through intelligent prompt distillation.
Summary updated