LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning

Shentong Mo, Yansen Wang, Dongsheng LiSubmission to ICLR 2024 · Rejected

TL;DR

Existing Visual Prompt Tuning (VPT) methods for Vision Transformers (ViTs) suffer from temporal and spatial forgetting: deeper transformer layers discard early prompt history and lose track of local spatial features. Long-term Spatial Prompt Tuning (LSPT) addresses both issues by feeding patch token averages into prompt representations and passing them across layers using a weight-shared LSTM cell. This simple, parameter-efficient approach significantly boosts downstream adaptation on both self-supervised and supervised ViTs across fine-grained classification, VTAB-1K, and semantic segmentation benchmarks.

What Problem Does This Paper Address?

Fine-tuning large pre-trained Vision Transformers (ViTs) on downstream tasks is computationally expensive. Visual Prompt Tuning (VPT) addresses this by freezing the backbone and learning only a small set of prependable tokens ("prompts").

However, existing VPT approaches (like standard VPT or Gated Prompt Tuning / GaPT) exhibit two major failure modes, particularly when adapting self-supervised backbones (e.g., MAE, MoCo v3):

  1. Temporal Forgetting: In deep architectures, newly injected prompts at deeper layers discard or exponentially decay information from early blocks, losing learned representations from earlier stages.
  2. Spatial Forgetting: Prompts often lack direct, persistent interaction with image patch tokens across layers, leading to degraded, blurry spatial attention maps in posterior transformer blocks.

Methods

LSPT draws inspiration from temporal and spatial coding in the biological visual cortex, introducing two lightweight mechanisms between consecutive transformer blocks:

[Transformer Block l]
   ├── Output Prompts (X̂_P) & Patch Tokens (X)
   │         ↓
   ├── [Class-aware Spatial Prompt Coding (CSPC)]
   │         ↓  (Add global average of patch tokens to prompts)
   │     Spatial Prompts (X̂_SP)
   │         ↓
   └── [Long-term Prompt Coding (LPC)]
             ↓  (LSTM step: input = injected prompt P^l, hidden = X̂_SP, cell = C^{l-1})
       Updated Prompts (X_P^l) ───> Fed into [Block l + 1]
  • Class-aware Spatial Prompt Coding (CSPC): At block $l$, the global average of the output patch embeddings is directly added to each output prompt token ($X_{SP,k}^l = X_{P,k}^l + \frac{1}{N} \sum_i X_i^l$). This injects global spatial context into the prompt representation without adding any trainable parameters.
  • Long-term Prompt Coding (LPC): Instead of discarding prior prompts or using only single-step Markovian gating, LSPT routes prompt updates through a single, parameter-shared Long Short-Term Memory (LSTM) layer across transformer layers. The newly inserted prompt $P^l$ acts as the input, the spatially augmented prompt $\hat{X}_{SP}^l$ acts as the hidden state, and the previous context $C^{l-1}$ serves as the cell state. This preserves long-range multi-block history and controls forgetting via LSTM gating.

Datasets

The authors evaluated LSPT across several standardized transfer learning benchmarks:

  • FGVC (Fine-Grained Visual Categorization): 5 datasets — CUB-200-2011, Oxford Flowers, Stanford Cars, Stanford Dogs, and NABirds.
  • VTAB-1K: 19 diverse image classification tasks partitioned into Natural (7), Specialized (4), and Structured (8) tasks, each with 1,000 training examples.
  • ADE20K: Semantic scene parsing/segmentation benchmark evaluated via the SETR-PUP framework.
  • Note: No new datasets were introduced; standard splits and protocols from prior literature were strictly followed.

Key Insights and Ideas

  • Recurrent connections resolve prompt amnesia: Treating the sequence of transformer blocks as a "temporal" sequence allows classical recurrent mechanisms (LSTM) to pass accumulated task-specific context across the entire network depth.
  • LSTMs outperform Transformers for cross-layer prompt aggregation: Ablation experiments showed that using a transformer layer instead of an LSTM to aggregate prompts degraded performance (75.89% vs. 76.94% average accuracy on FGVC). Self-attention among prompts causes tokens to homogenize toward similar values, whereas an LSTM's explicit forget gate maintains representation variance.
  • Global patch pooling preserves localization: Simply adding the mean patch embedding to prompt tokens prevents attention collapse in later layers, yielding sharply focused attention maps around downstream target objects in layer 12 (as visualized in Figures 1 and 5).

Key Results

  • Self-Supervised Adaptation (FGVC & VTAB-1K):
  • On MAE ViT-B/16, LSPT achieved 76.94% average accuracy on FGVC (vs. 73.39% for GaPT and 72.02% for VPT-Deep) and 53.86% on VTAB-1K (vs. 49.22% for GaPT).
  • On MoCo v3 ViT-B/16, LSPT achieved 84.18% on FGVC (vs. 83.00% for GaPT) and 68.72% on VTAB-1K (vs. 65.80% for GaPT).
  • Supervised Transfer (ImageNet-21K ViT-B/16 on VTAB-1K): LSPT reached 77.95% overall average accuracy, outperforming Bi-AdaptFormer (74.73%) and VPT-Deep (69.42%).
  • Semantic Segmentation (ADE20K): Improved MAE multi-scale mIoU to 41.51% (compared to 39.81% for GaPT and 38.80% for VPT-Deep).
  • Computational Overhead: Compared to GaPT, LSPT adds negligible overhead (max memory: 24.02 GB vs. 23.78 GB; inference time per batch: 0.0872s vs. 0.0871s).

Limitations and Future Work

  • Heuristic Spatial Aggregation: CSPC relies primarily on global average pooling across patch tokens. While an exploratory $k$-means clustering alternative improved accuracy slightly (+0.23% on FGVC), it introduced significant training latency. Developing adaptive, non-uniform spatial token selection remains an open direction.
  • Single-Modality Focus: The current study evaluates purely vision-based pre-training and downstream transfer. The authors suggest extending long-term spatial/temporal prompt coding to multimodal vision-language setups (e.g., CLIP-style models).

Summary updated

Abstract

Visual Prompt Tuning (VPT) techniques have gained prominence for their capacity to adapt pre-trained Vision Transformers (ViTs) to downstream visual tasks using specialized learnable tokens termed as prompts. Contemporary VPT methodologies, especially when employed with self-supervised vision transformers, often default to the introduction of new learnable prompts or gated prompt tokens predominantly sourced from the model's previous block. A pivotal oversight in such approaches is their failure to harness the potential of long-range previous blocks as sources of prompts within each self-supervised ViT. To bridge this crucial gap, we introduce Long-term Spatial Prompt Tuning (LSPT) – a revolutionary approach to visual representation learning. Drawing inspiration from the intricacies of the human brain, LSPT ingeniously incorporates long-term gated prompts. This feature serves as temporal coding, curbing the risk of forgetting parameters acquired from earlier blocks. Further enhancing its prowess, LSPT brings into play patch tokens, serving as spatial coding. This is strategically designed to perpetually amass class-conscious features, thereby fortifying the model's prowess in distinguishing and identifying visual categories. To validate the efficacy of our proposed method, we engaged in rigorous experimentation across 5 FGVC and 19 VTAB-1K benchmarks. Our empirical findings underscore the superiority of LSPT, showcasing its ability to set new benchmarks in visual prompt tuning performance.
BibTeX
@misc{mo2024lspt,
  author = {Shentong Mo and Yansen Wang and Dongsheng Li},
  title = {{LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning}},
  year = {2024},
  eprint = {2402.17406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.17406},
}
Figure