TL;DR
This paper introduces CoPrompt, a novel method for fine-tuning large vision-language models (VLMs) in few-shot settings. CoPrompt enhances generalization by enforcing consistency between the model's predictions and its pre-trained self, using perturbed inputs and combining prompt and adapter tuning techniques. It achieves state-of-the-art results on various generalization benchmarks, significantly improving performance without sacrificing zero-shot capabilities.
What Problem Does This Paper Address?
Large foundation models like CLIP have impressive generalization abilities, but fine-tuning them for specific downstream tasks, especially with limited data (few-shot learning), is challenging. A major hurdle is preventing "overfitting" during this fine-tuning process. When new parameters are introduced, they can deviate too much from the original model's behavior, leading to a loss of the model's inherent generalization capabilities, particularly in zero-shot scenarios. Existing methods often struggle to balance improved few-shot performance with maintained or enhanced zero-shot abilities.
Methods
CoPrompt tackles this by introducing a consistency constraint. The core idea is to ensure that the fine-tuned model's outputs remain "close" to those of the original, pre-trained model. This is achieved by minimizing the cosine distance between their embeddings.
To further bolster generalization, CoPrompt incorporates two key innovations:
- Input Perturbation: Instead of comparing predictions on the exact same input, CoPrompt uses slightly altered versions. For text, a pre-trained Large Language Model (LLM) like GPT generates more descriptive sentences from a generic prompt. For images, data augmentations create perturbed versions. Enforcing consistency on these perturbed inputs acts as a stronger regularizer.
- Prompting and Adapters: CoPrompt combines two popular parameter-efficient tuning techniques:
- Prompt Tuning: Learns continuous "prompts" (vectors) that guide the model's attention.
- Adapters: Small, trainable modules inserted into the model's layers to transform feature representations. By using both, CoPrompt can tune more parameters, potentially improving task-specific performance, while the consistency constraint prevents detrimental deviations.
The final loss function is a combination of a standard supervised loss and the proposed consistency loss, balanced by a weighting factor.
Datasets
The paper evaluates CoPrompt on several benchmark tasks:
- Base-to-Novel Generalization: Evaluated on 11 recognition datasets.
- Cross-Dataset Evaluation: Models are fine-tuned on ImageNet and evaluated on target datasets in a zero-shot manner.
- Domain Generalization: Models are fine-tuned on the original ImageNet dataset and tested on variants with different distributions (ImageNetV2, ImageNet-Sketch, ImageNet-A, ImageNet-R).
The paper does not introduce new datasets but uses established ones to demonstrate performance across various generalization challenges.
Key Insights and Ideas
- Consistency is Key for Generalization: The central insight is that enforcing consistency between the fine-tuned and pre-trained model's representations is crucial for preventing overfitting and maintaining/improving zero-shot generalization.
- Perturbations Add Robustness: Using perturbed inputs for consistency checks acts as a more effective regularizer than using identical inputs, leading to better generalization.
- Synergy of Prompting and Adapters: Combining prompt tuning and adapters allows for more parameter tuning, which can boost performance, without compromising the benefits of the consistency constraint.
- No Trade-off Between Few-Shot and Zero-Shot: CoPrompt demonstrates that it's possible to improve few-shot performance and zero-shot generalization simultaneously, a significant challenge in prior work.
Key Results
- Base-to-Novel Generalization: CoPrompt achieves state-of-the-art results, improving zero-shot performance on novel classes by 2.09% and the harmonic mean by 1.93% over the previous SOTA (MaPLe). It also improves performance on base classes by 1.72%.
- Cross-Dataset Evaluation: CoPrompt shows improvements on 9 out of 10 datasets, achieving an average accuracy of 67.0%, which is 0.70% higher than MaPLe.
- Domain Generalization: CoPrompt demonstrates strong domain generalization, outperforming MaPLe on most variants of ImageNet and achieving an average accuracy of 60.43%.
- Ablation Studies: Removing components like adapters or input perturbations leads to performance drops, confirming the effectiveness of each part of CoPrompt. Notably, even the consistency constraint alone provides a significant improvement over training without it.
Limitations and Future Work
- Computational Cost: While CoPrompt uses parameter-efficient tuning, the use of an LLM for text perturbation might add computational overhead.
- Hyperparameter Tuning: The balancing factor ($\lambda$) for the consistency loss is a hyperparameter that needs careful tuning.
The paper suggests that the proposed consistency-guided tuning approach has broad applicability for tuning vision and vision-language models across various applications. Future work could explore different consistency metrics, more sophisticated perturbation strategies, and applying CoPrompt to other foundation model architectures.
Summary updated