SA²VP: Spatially Aligned-and-Adapted Visual Prompt

Wenjie Pei, Tongqi Xia, Fanglin Chen, Jinsong Li, Jiandong Tian, Guangming LuAAAI 2024 · DOI

Abstract

SA²VP reimagines visual prompt tuning by replacing the standard "sequence of tokens" approach with a 2D prompt map that spatially aligns with the input image. This allows for fine-grained, location-specific prompting and better preservation of spatial relationships, leading to state-of-the-art performance across multiple computer vision benchmarks with minimal parameter overhead.

Introduction

Parameter-Efficient Fine-Tuning (PEFT) has become the gold standard for adapting massive pre-trained models to specific tasks without retraining billions of parameters. In computer vision, Visual Prompt Tuning (VPT) is a popular PEFT method that adds a small set of learnable "prompt" tokens to the input.

However, most VPT methods borrow their architecture directly from Natural Language Processing (NLP), treating an image as a flat, 1D sequence of tokens. This "sequential modeling" paradigm suffers from two major flaws: it fails to capture the underlying 2D spatial relationships inherent in images, and it lacks precision, as every prompt token interacts with every image token equally regardless of spatial relevance.

Comparison between the traditional sequential modeling paradigm and the proposed spatially aligned SA²VP.

Methodology

The authors propose the Spatially Aligned-and-Adapted Visual Prompt (SA²VP), which introduces several novel components to restore spatial inductive bias to the prompting process:

  • 2D Prompt Map: Instead of unordered tokens, SA²VP learns a 2D grid of prompt tokens matching the image's patch grid. Pre-trained positional embeddings ensure the model preserves the spatial structure of the prompt.
  • Siamese Dual-Pathway Architecture: The model runs two parallel pathways—a Prompt Pathway and a Base Pathway—both utilizing the same frozen pre-trained backbone (e.g., a Vision Transformer).
  • Bilateral Interaction (BI): The pathways communicate via a "Spatially-Aligned Bidirectional Cross-Attention" mechanism. A token at a specific $(x, y)$ coordinate in one pathway only attends to a small local window around the same coordinate in the other pathway, ensuring prompts only influence spatially corresponding image regions.
  • Prompt Adapter & Fusion: A lightweight bottleneck MLP adapts prompted features, while Inter-layer Prompt Fusion captures multi-scale features by learning prompt maps at different depths of the backbone.
Detailed architecture of the Bilateral Interaction (BI) module and the overall dual-pathway pipeline.

Experiments and Results

SA²VP was evaluated on three major benchmarks: FGVC (Fine-Grained Visual Classification), HTA (Head Tuning Adaptation), and VTAB-1k (a diverse 19-dataset benchmark).

SA²VP consistently outperformed existing state-of-the-art methods like VPT-deep, E²VPT, and AdaptFormer while tuning less than 1% of the total parameters:

  • FGVC: Achieved a mean accuracy of 90.08%, surpassing VPT-deep (89.11%).
  • HTA: Reached a mean of 91.5%, notably outperforming DAM-VP (88.5%).
  • VTAB-1k: On challenging "Structured" tasks (geometric reasoning), SA²VP hit 60.80% accuracy, a significant jump over VPT-deep's 54.98%.

Qualitative analysis via t-SNE visualizations further confirmed that SA²VP produces tighter, more distinct class clusters, indicating higher-quality feature learning.

Discussion and Key Insights

The core innovation of SA²VP is the recognition that images are not sentences. Spatial alignment is a fundamental inductive bias for vision. By restricting prompt interaction to local spatial windows, the model effectively regularizes the learning process, which is particularly beneficial for complex scenes and geometric reasoning tasks.

While SA²VP is highly parameter-efficient, its dual-pathway architecture increases the computational footprint (FLOPs) during the forward pass. Future work may explore more efficient implementations of the dual-pathway interaction to mitigate this overhead while maintaining the performance gains of spatial alignment.

Summary updated

BibTeX
@inproceedings{pei2024sa,
  author = {Wenjie Pei and Tongqi Xia and Fanglin Chen and Jinsong Li and Jiandong Tian and Guangming Lu},
  title = {{SA²VP: Spatially Aligned-and-Adapted Visual Prompt}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i5.28243},
}
Figure