E2VPT: An Effective and Efficient Approach for Visual Prompt Tuning.

Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, Dongfang LiuICCV 2023 · DOI · OpenReview

TL;DR

This paper introduces E2VPT, a novel parameter-efficient visual prompt tuning method that significantly improves adaptation of large pre-trained vision transformers to new tasks. By incorporating learnable key-value prompts into the self-attention mechanism and employing a prompt pruning strategy, E2VPT achieves state-of-the-art performance with a fraction of the tunable parameters compared to existing methods.

What Problem Does This Paper Address?

The rapid growth in the size of pre-trained vision models, particularly transformers, has made fine-tuning them for downstream tasks computationally expensive and parameter-intensive. While parameter-efficient fine-tuning (PEFT) methods aim to reduce this burden, they often suffer from a performance gap compared to full fine-tuning. Existing prompt tuning methods, for instance, typically only modify the input sequence and overlook the internal workings of the transformer's self-attention mechanism, limiting their effectiveness. Furthermore, achieving extreme parameter efficiency without sacrificing performance remains a challenge.

Methods

E2VPT addresses these issues through two main innovations:

  • Effective Prompting with Key-Value Prompts: Unlike traditional prompt tuning that only prepends prompts to the input, E2VPT introduces learnable "key-value" prompts (PK and PV) that are incorporated directly into the self-attention module's key and value matrices. This allows the model to better adapt its attention mechanism to new data and learn task-specific patterns. The visual prompts (P1) are still prepended to the input sequence.
  • Efficient Prompt Pruning: To further reduce the number of tunable parameters, E2VPT employs a two-level pruning strategy inspired by the lottery ticket hypothesis.
  • Token-wise Pruning: This step identifies and removes individual visual prompt tokens that have low importance scores, calculated based on their expected sensitivity to the model's loss.
  • Segment-wise Pruning: This further refines the pruning by dividing remaining prompt tokens into segments and removing those with low importance.
  • Rewinding: After pruning, the remaining "soft-filtered" prompts are re-trained along with the backbone.

Datasets

The paper evaluates E2VPT on two primary image classification benchmarks:

  • VTAB-1k: This benchmark comprises 19 diverse tasks categorized into Natural, Specialized, and Structured image sets.
  • FGVC: This benchmark includes 5 fine-grained visual classification tasks.

Experiments were also conducted using models pretrained with different objectives:

  • ImageNet-21k (supervised): The primary pretraining dataset for ViT and Swin transformers.
  • MAE (Masked Autoencoders): A self-supervised pretraining method.
  • MoCo v3: Another self-supervised pretraining method.

Key Insights and Ideas

  • Self-attention is Key: The paper strongly argues that adapting the self-attention mechanism itself, not just the input, is crucial for effective prompt tuning. The introduction of key-value prompts directly targets this.
  • Pruning for Efficiency: The lottery ticket hypothesis provides a strong theoretical basis for pruning prompts. The idea that a sparse sub-network can perform as well as a dense one is applied to prompt tokens, leading to significant parameter reduction.
  • Robustness to Prompt Location: The paper shows that placing key-value prompts before or after the key and value matrices in the self-attention layer yields similar competitive results, indicating robustness in the architectural design.
  • Hyperbolic Visualization: The use of hyperbolic visualizations effectively demonstrates that E2VPT leads to better-separated and clustered embeddings, indicating improved discriminative power for new tasks.

Key Results

  • Performance Gains: E2VPT outperforms several state-of-the-art baselines on VTAB-1k and FGVC benchmarks. It achieves an average improvement of 5.85% on VTAB-1k compared to full fine-tuning and 1.99% compared to VPT.
  • Parameter Efficiency: E2VPT uses significantly fewer tunable parameters, averaging only 0.32% of the model parameters on VTAB-1k, compared to VPT's average of 0.68%. For instance, on ViT-Base/16, E2VPT tunes only 0.39% of parameters.
  • Generalization: The method demonstrates strong performance across different backbone architectures (ViT, Swin) and pretraining objectives (supervised, MAE, MoCo v3).
  • Component Impact: Ablation studies show that visual prompts, key-value prompts, and the pruning mechanism all contribute to performance gains, with the combined approach yielding the best results.

Limitations and Future Work

  • Prompt Length Sensitivity: The paper notes that while E2VPT is robust to prompt location, excessively long key-value prompts can lead to subpar results, suggesting a need for careful hyperparameter tuning.
  • Pruning in Deeper Layers: For hierarchical transformers like Swin, performance drops were observed when applying pruning to deeper local windows, suggesting that pruning strategies might need to be adapted for different architectural components.
  • Algorithmic Generalization: The authors mention demonstrating the approach's generalization to the language domain in the Appendix, hinting at potential cross-modal applications.
  • Further Exploration: The paper suggests that further investigation into prompt placement and the impact of different initialization methods is warranted.

Summary updated

BibTeX
@inproceedings{han2023e,
  author = {Cheng Han and Qifan Wang and Yiming Cui and Zhiwen Cao and Wenguan Wang and Siyuan Qi and Dongfang Liu},
  title = {{E2VPT: An Effective and Efficient Approach for Visual Prompt Tuning.}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01604},
}
Figure