Learning Through Consistency for Prompt Tuning

Shuvendu Roy, Ali EtemadR0-FoMo @ NeurIPS 2023 · OpenReview

TL;DR

This paper introduces CoPrompt, a novel method for fine-tuning large vision-language models (VLMs) in few-shot settings. CoPrompt enhances generalization by enforcing consistency between the model's predictions and its pre-trained self, using perturbed inputs and combining prompt and adapter tuning techniques. It achieves state-of-the-art results on various generalization benchmarks, significantly improving performance without sacrificing zero-shot capabilities.

What Problem Does This Paper Address?

Large foundation models like CLIP have impressive generalization abilities, but fine-tuning them for specific downstream tasks, especially with limited data (few-shot learning), is challenging. A major hurdle is preventing "overfitting" during this fine-tuning process. When new parameters are introduced, they can deviate too much from the original model's behavior, leading to a loss of the model's inherent generalization capabilities, particularly in zero-shot scenarios. Existing methods often struggle to balance improved few-shot performance with maintained or enhanced zero-shot abilities.

Methods

CoPrompt tackles this by introducing a consistency constraint. The core idea is to ensure that the fine-tuned model's outputs remain "close" to those of the original, pre-trained model. This is achieved by minimizing the cosine distance between their embeddings.

To further bolster generalization, CoPrompt incorporates two key innovations:

  1. Input Perturbation: Instead of comparing predictions on the exact same input, CoPrompt uses slightly altered versions. For text, a pre-trained Large Language Model (LLM) like GPT generates more descriptive sentences from a generic prompt. For images, data augmentations create perturbed versions. Enforcing consistency on these perturbed inputs acts as a stronger regularizer.
  2. Prompting and Adapters: CoPrompt combines two popular parameter-efficient tuning techniques:
  3. Prompt Tuning: Learns continuous "prompts" (vectors) that guide the model's attention.
  4. Adapters: Small, trainable modules inserted into the model's layers to transform feature representations. By using both, CoPrompt can tune more parameters, potentially improving task-specific performance, while the consistency constraint prevents detrimental deviations.

The final loss function is a combination of a standard supervised loss and the proposed consistency loss, balanced by a weighting factor.

Datasets

The paper evaluates CoPrompt on several benchmark tasks:

  • Base-to-Novel Generalization: Evaluated on 11 recognition datasets.
  • Cross-Dataset Evaluation: Models are fine-tuned on ImageNet and evaluated on target datasets in a zero-shot manner.
  • Domain Generalization: Models are fine-tuned on the original ImageNet dataset and tested on variants with different distributions (ImageNetV2, ImageNet-Sketch, ImageNet-A, ImageNet-R).

The paper does not introduce new datasets but uses established ones to demonstrate performance across various generalization challenges.

Key Insights and Ideas

  • Consistency is Key for Generalization: The central insight is that enforcing consistency between the fine-tuned and pre-trained model's representations is crucial for preventing overfitting and maintaining/improving zero-shot generalization.
  • Perturbations Add Robustness: Using perturbed inputs for consistency checks acts as a more effective regularizer than using identical inputs, leading to better generalization.
  • Synergy of Prompting and Adapters: Combining prompt tuning and adapters allows for more parameter tuning, which can boost performance, without compromising the benefits of the consistency constraint.
  • No Trade-off Between Few-Shot and Zero-Shot: CoPrompt demonstrates that it's possible to improve few-shot performance and zero-shot generalization simultaneously, a significant challenge in prior work.

Key Results

  • Base-to-Novel Generalization: CoPrompt achieves state-of-the-art results, improving zero-shot performance on novel classes by 2.09% and the harmonic mean by 1.93% over the previous SOTA (MaPLe). It also improves performance on base classes by 1.72%.
  • Cross-Dataset Evaluation: CoPrompt shows improvements on 9 out of 10 datasets, achieving an average accuracy of 67.0%, which is 0.70% higher than MaPLe.
  • Domain Generalization: CoPrompt demonstrates strong domain generalization, outperforming MaPLe on most variants of ImageNet and achieving an average accuracy of 60.43%.
  • Ablation Studies: Removing components like adapters or input perturbations leads to performance drops, confirming the effectiveness of each part of CoPrompt. Notably, even the consistency constraint alone provides a significant improvement over training without it.

Limitations and Future Work

  • Computational Cost: While CoPrompt uses parameter-efficient tuning, the use of an LLM for text perturbation might add computational overhead.
  • Hyperparameter Tuning: The balancing factor ($\lambda$) for the consistency loss is a hyperparameter that needs careful tuning.

The paper suggests that the proposed consistency-guided tuning approach has broad applicability for tuning vision and vision-language models across various applications. Future work could explore different consistency metrics, more sophisticated perturbation strategies, and applying CoPrompt to other foundation model architectures.

Summary updated

Abstract

We propose Consistency-guided Prompt learning (CoPrompt), a new fine-tuning method for vision-language models that addresses the challenge of improving the generalization capability of large foundation models while fine-tuning them on downstream tasks in a few-shot setting. The basic idea of CoPrompt is to enforce a consistency constraint in the prediction of the trainable and pre-trained models to prevent overfitting on the downstream task. Additionally, we introduce the following two components into our consistency constraint to further boost the performance: enforcing consistency on two perturbed inputs and combining two dominant paradigms of tuning, prompting and adapter. Enforcing consistency on perturbed input further regularizes the consistency constraint, effectively improving generalization, while tuning additional parameters with prompting and adapters improves the performance on downstream tasks. Extensive experiments show that CoPrompt outperforms existing methods on a range of evaluation suites, including base-to-novel generalization, domain generalization, and cross-dataset evaluation tasks. On the generalization task, CoPrompt improves the state-of-the-art by 2.09\% on the zero-shot task and 1.93\% on the harmonic mean over 11 recognition datasets. Detailed ablation studies show the effectiveness of each of the components in CoPrompt.
BibTeX
@inproceedings{roy2023learning,
  author = {Shuvendu Roy and Ali Etemad},
  title = {{Learning Through Consistency for Prompt Tuning}},
  booktitle = {R0-FoMo @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=r7YtqrcPQN},
}
Figure