Multi-Vision Multi-Prompt for Few-Shot Learning in Vision-Language Model
Lists
Only you see this
Abstract
In vision-language models such as the Contrastive Language-Image Pre-Training model (CLIP), prompt learning can efficiently and rapidly adapt to specific tasks in few-shot learning. Previous methods for prompt learning often rely on a single prompt. However, a single prompt may not accurately distinguish between different categories, especially when a category has multiple features and contextual connections in a few-shot learning environment. While the performance of few-shot learning can improve through meta-learning or image augmentation strategies, these approaches may increase computational cost and affect accuracy. To address these issues, we propose a new method called Multi-Vision Multi-Prompt (MVMP), designed for CLIP in a few-shot learning environment. Instead of increasing the number of model parameters, MVMP employs multiple prompts at different stages of the training process and averages the predictions. Additionally, we present a mixed self-augmentation framework and text distillation to further enhance the model's performance. Extensive experimental validation demonstrates that our approach significantly outperforms the state-of-the-art in the few-shot learning classification tasks, improving accuracy by 4.6% and 2%.
BibTeX
@misc{wang2024multi,
author = {Ran Wang and Hua Zuo and Zhen Fang and Jie Lu},
title = {{Multi-Vision Multi-Prompt for Few-Shot Learning in Vision-Language Model}},
year = {2024},
url = {https://openreview.net/forum?id=j1FLTvgyAh},
howpublished = {OpenReview},
note = {Submitted to ICLR 2024},
}Loading related content...