TL;DR
MoPD (Mixture-of-Prompts Distillation) improves how Vision-Language Models (VLMs) like CLIP adapt to new tasks by distilling knowledge from a diverse pool of hand-crafted "hard" prompts into learnable "soft" prompts. By using an image-conditioned gating network to select the best teacher prompts for every specific image, MoPD significantly boosts performance on unseen classes without sacrificing accuracy on seen ones.
What Problem Does This Paper Address?
Large-scale Vision-Language Models (VLMs) like CLIP are famous for their "zero-shot" abilities—they can recognize objects they weren't explicitly trained on by using natural language descriptions (hard prompts) like "a photo of a [CLASS]."
To make these models even better at specific tasks, researchers developed Soft Prompt Learning (e.g., CoOp). Instead of hand-writing prompts, they let the model learn optimal "prompt vectors." While this works wonders for the specific classes seen during training (base classes), it often backfires: the model "overfits" to those classes and loses its ability to generalize to new, unseen classes.
The challenge is: how do we keep the flexibility of learnable soft prompts while retaining the robust generalization of human-designed hard prompts?
Methods
The authors introduce MoPD (Mixture-of-Prompts Distillation), which treats the learning process as a teacher-student problem.
- The Student: A learnable soft prompt (similar to CoOp).
- The Teachers: A pool of multiple hand-crafted hard prompts (e.g., "a type of texture," or "an image labeled as [CLASS]").
- The Gating Network: This is the "secret sauce." Inspired by Mixture-of-Experts (MoE), the authors add a lightweight gating network that looks at an input image and decides which hard prompts from the pool are the most relevant "teachers" for that specific image.
- The Distillation Loss: The model is trained using a multi-part objective:
- Cross-Entropy Loss: Standard classification accuracy.
- Distillation Loss ($\zeta_{MPD}$): Minimizes the KL divergence between the soft prompt's prediction and the weighted average of the selected hard prompts' predictions.
- Selection Loss ($\zeta_{MPS}$): Encourages the gating network to assign higher weights to hard prompts that actually yield correct predictions for the current image.
Crucially, the gating network is only used during training. At inference time, the model uses the optimized soft prompt alone, making it just as fast as standard methods.
Datasets
The researchers evaluated MoPD across 11 diverse image classification datasets:
- Generic Objects: ImageNet, Caltech101.
- Fine-grained Categories: OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft.
- Specialized Domains: SUN397 (scenes), DTD (textures), EuroSAT (satellite), UCF101 (actions).
They also tested Domain Generalization using ImageNet variants (V2, Sketch, A, and R) to see how the model handles style shifts.
Key Insights and Ideas
- More Teachers are Better than One: The paper demonstrates that distilling from a single hard prompt is helpful, but distilling from a mixture of prompts is significantly more effective. Different prompts capture different nuances of the VLM's pre-trained knowledge.
- Instance-Specificity Matters: Not every image "prefers" the same prompt. An image with a busy background might benefit from a different prompt template than a clean studio shot. The gating network allows the model to dynamically choose the best guidance for each training sample.
- Theoretical Grounding in MoE: The authors link their approach to the "Mixture-of-Experts" paradigm, noting that while a single "expert" (prompt) might perform poorly, a router that divides a complex problem into simpler sub-problems (handled by different prompts) can provably outperform a single expert.
- Robustness to Noise: Interestingly, the gating network is smart enough to ignore "noisy" or irrelevant prompts added to the pool, effectively filtering out bad advice from the teacher pool.
Key Results
MoPD was tested primarily on the Base-to-New Generalization task, where the model is trained on one set of classes and tested on a completely different set.
- Superior Generalization: MoPD achieved the highest Harmonic Mean (H) score (a metric balancing base and new class accuracy) in 10 out of 11 datasets compared to other textual prompting methods.
- Average Gains: Across all 11 datasets, MoPD improved the accuracy on "New" classes by a significant margin over the baseline CoOp (74.69% vs 68.00%).
- Efficiency: While multi-modal methods like MaPLe perform well, they use millions of tunable parameters. MoPD achieves competitive or better results with only ~8K parameters, making it much lighter to train.
- Domain Robustness: On ImageNet variants, MoPD-MMP (a multi-modal extension) achieved an overall average accuracy of 60.63%, outperforming all established baselines.
Limitations and Future Work
- The "OxfordPets" Exception: MoPD slightly underperformed on the OxfordPets dataset. The authors suggest this is due to a very small "category shift" between CLIP's original training data and this specific task, meaning the hard prompts provided less useful "new" knowledge to distill.
- Prompt Pool Design: While MoPD is robust to noise, it still relies on a human-defined pool of hard prompts. Future work could explore using Large Language Models (LLMs) to automatically generate even more diverse and effective teacher prompt pools.
- Gating Complexity: The current gating network is a simple linear layer. Exploring more sophisticated architectures for the "prompt router" could yield further gains.
Summary updated