Multi-Vision Multi-Prompt for Few-Shot Learning in Vision-Language Model

Ran Wang, Hua Zuo, Zhen Fang, Jie LuOpenReview 2024 · Submission to ICLR 2024 · Rejected · OpenReview

Abstract

In vision-language models such as the Contrastive Language-Image Pre-Training model (CLIP), prompt learning can efficiently and rapidly adapt to specific tasks in few-shot learning. Previous methods for prompt learning often rely on a single prompt. However, a single prompt may not accurately distinguish between different categories, especially when a category has multiple features and contextual connections in a few-shot learning environment. While the performance of few-shot learning can improve through meta-learning or image augmentation strategies, these approaches may increase computational cost and affect accuracy. To address these issues, we propose a new method called Multi-Vision Multi-Prompt (MVMP), designed for CLIP in a few-shot learning environment. Instead of increasing the number of model parameters, MVMP employs multiple prompts at different stages of the training process and averages the predictions. Additionally, we present a mixed self-augmentation framework and text distillation to further enhance the model's performance. Extensive experimental validation demonstrates that our approach significantly outperforms the state-of-the-art in the few-shot learning classification tasks, improving accuracy by 4.6% and 2%.
BibTeX
@misc{wang2024multi,
  author = {Ran Wang and Hua Zuo and Zhen Fang and Jie Lu},
  title = {{Multi-Vision Multi-Prompt for Few-Shot Learning in Vision-Language Model}},
  year = {2024},
  url = {https://openreview.net/forum?id=j1FLTvgyAh},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}

Loading related content...

Figure