Attention to the Burstiness in Visual Prompt Tuning!

Yuzhu Wang, Manni Duan, Shu Kong

TL;DR

This paper reveals that standard Visual Prompt Tuning (VPT) struggles because the data distributions inside a Vision Transformer's (ViT) self-attention module are highly non-Gaussian and "bursty" (dominated by a few extreme values). To fix this, the authors introduce Bilinear Prompt Tuning (BPT), a method that uses data whitening and low-rank bilinear factorization to make prompt learning significantly faster, more accurate, and computationally efficient.

What Problem Does This Paper Address?

Visual Prompt Tuning (VPT) is a popular, parameter-efficient method to adapt large pre-trained Vision Transformers (ViTs) to new downstream tasks. Instead of fine-tuning billions of parameters, VPT simply prepends a small set of learnable parameters—called "prompts"—to the input image patches.

However, learning these prompts is notoriously difficult. The authors investigate why and uncover a hidden culprit in the self-attention mechanism: burstiness and non-Gaussian data distributions.

Specifically, they find that the values resulting from the interaction of image patch embeddings ($X$) and the key/query projectors ($W_q, W_k$) follow heavy-tailed Laplacian and hyper-Laplacian distributions rather than a standard Gaussian distribution. Within the attention module, a tiny fraction of entries end up with massive absolute values (a phenomenon called burstiness), as illustrated in the distributions of Figure 1. This statistical mismatch creates a harsh optimization landscape, making it hard for standard prompts to converge effectively.

Visualizing burstiness and non-Gaussian distributions in ViT attention modules

Methods

To counteract these problematic distributions, the authors propose Bilinear Prompt Tuning (BPT), exploring three distinct flavors based on data whitening and bilinear modeling:

  1. BPT-fWhiten (Fixed Whitening): The authors apply Zero-phase Component Analysis (ZCA) whitening to the patch embeddings and attention projectors using a small sample of training images. This decorrelates the features and equalizes their variance, transforming the distribution to be "more Gaussian." The prompt is multiplied by this fixed whitening matrix $W$, yielding an effective prompt $\tilde{P} = PW^T$.
  2. BPT-tWhiten (Tunable Whitening): Recognizing that the Transformer is differentiable with respect to the whitening matrix, this variant allows $W$ to be fine-tuned alongside the prompt $P$ during training.
  3. BPT-bilinear (Low-Rank Bilinear Prompts): Since the whitening approach naturally results in a two-matrix multiplication (a bilinear form), the authors take it a step further. They replace the whitening matrix entirely with a low-rank factorization, learning two compact matrices $A$ and $B$ such that the final prompt is $\tilde{P} = AB^T$. This variant requires no data preprocessing and drastically reduces parameter count.

The conceptual differences between these three approaches are mapped out in Figure 2.

Conceptual overview of BPT-fWhiten, BPT-tWhiten, and BPT-bilinear

Datasets

The authors evaluated BPT across a diverse set of computer vision benchmarks:

  • Fine-Grained Visual Classification (FGVC): CUB-200-2011, NABirds, Oxford Flowers, Stanford Dogs, and Stanford Cars.
  • General Image Classification: ImageNet-1K.
  • Object Detection & Instance Segmentation: COCO 2017 (using the ViTDet framework).

Key Insights and Ideas

  • The "Bursty Prompt" Paradox: Intuitively, whitening is used to make the input data less bursty and more Gaussian. Surprisingly, when the authors looked at the learned effective prompts ($\tilde{P}$) produced by BPT, they found that the prompts themselves became highly bursty (containing a few very large values). This suggests that Transformers actually prefer interacting with bursty prompts, and forcing the input data to be more Gaussian gives the model the freedom to learn these highly effective, bursty prompt structures.
  • Data Scale Efficiency: Computing the fixed whitening matrix doesn't require massive compute. The authors found that sampling as few as 100 random training images was more than enough to calculate an effective whitening matrix $W$.
  • Bilinear Factorization as a Regularizer: Moving to a low-rank bilinear structure ($AB^T$) acts as an excellent regularizer. It eliminates the need for careful prompt initialization (which methods like SPT require) and makes prompt learning robust to random initialization.

Key Results

BPT achieves massive performance leaps over standard VPT and other state-of-the-art parameter-efficient fine-tuning (PEFT) methods:

  • Massive Accuracy Gains: On the challenging CUB-200 fine-grained dataset, standard VPT achieves a meager 42.15% accuracy. BPT-fWhiten skyrockets this to 77.48%—a massive 35-point absolute improvement.
  • Computation and Parameter Reduction: BPT-bilinear achieves a 3.5$\times$ reduction in computation overhead compared to standard VPT by leveraging its low-rank matrix structure, all while using fewer parameters and delivering superior accuracy.
  • Robustness to Scale: As shown in Figure 4, BPT consistently outperforms competing methods like SPT across different backbone scales (from 86M to 2B parameters), training schedules, and data sizes.
Performance scaling across backbone sizes, training schedules, and data sizes

Limitations and Future Work

While BPT delivers impressive empirical results, the authors note a few limitations:

  • Simplified Analysis: The mathematical motivation focuses primarily on a single interaction term ($PW_qW_k^TX^T$) within the self-attention formula, omitting other interacting terms for simplicity. Future work could benefit from a more holistic analysis.
  • Theoretical Dynamics: The explanation relies heavily on empirical observations of burstiness. A deeper theoretical exploration of the learning dynamics of bilinear prompts would strengthen the foundational understanding of why this works so well.
  • Network-Wide Burstiness: This study limits its view to the prompt interaction layer, but burstiness likely propagates throughout the deeper layers of the Transformer network. Investigating and mitigating this network-wide phenomenon remains an open research direction.

Summary updated

Abstract

Visual Prompt Tuning (VPT) is a parameter-efficient fune-tuning technique that adapts a pre-trained vision Transformer (ViT) by learning a small set of parameters in the input space, known as prompts. In VPT, we uncover ``burstiness'' in the values arising from the interaction of image patch embeddings, and the key and query projectors within Transformer's self-attention module. Furthermore, the values of patch embeddings and the key and query projectors exhibit Laplacian and hyper-Laplacian distribution, respectively. Intuitively, these non-Gaussian distributions pose challenges for learning prompts. To address this, we propose whitening these data, de-correlating them and equalizing their variance towards more Gaussian before learning prompts. We derive the whitening matrix over random image patch embeddings and ViT's key and query projectors, and multiply it with the prompt to be learned in a bilinear manner. Surprisingly, this method significantly accelerates prompt tuning and boosts accuracy, e.g., $>$25 accuracy points on the CUB dataset; interestingly, it learns ``bursty prompts''. Extending the bilinear model which is known to introduce burstiness, we present a compact, low-rank version by learning two smaller matrices whose multiplication yields the final prompts. We call the proposed methods Bilinear Prompt Tuning (BPT). Extensive experiments across multiple benchmark datasets demonstrate that BPT methods not only outperform various VPT methods but also reduce parameter count and computation overhead.
BibTeX
@misc{wang2025attention,
  author = {Yuzhu Wang and Manni Duan and Shu Kong},
  title = {{Attention to the Burstiness in Visual Prompt Tuning!}},
  year = {2025},
  eprint = {2506.22908},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.22908},
}
Figure