LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning

Shentong Mo, Yansen Wang, Xufang Luo, Dongsheng Li

Introduction

Adapting large-scale, self-supervised Vision Transformers (ViTs) to specific downstream tasks is a computationally intensive process. While Visual Prompt Tuning (VPT) offers a parameter-efficient alternative by training only a small set of prompt tokens while keeping the backbone frozen, it suffers from two significant "forgetting" phenomena:

  1. Temporal Forgetting: Information from early transformer blocks decays exponentially as it propagates through the network, leading to a loss of long-term context in deeper layers.
  2. Spatial Forgetting: Prompts frequently lose their structural connection to image patch tokens, resulting in degraded spatial awareness and "blurry" attention maps.

LSPT (Long-term Spatial Prompt Tuning) addresses these issues by integrating a long-term memory mechanism and global spatial coding, setting new performance benchmarks across diverse vision datasets while remaining highly parameter-efficient.

Methodology

LSPT introduces two core modules designed to mitigate information loss and maintain structural grounding:

Global Spatial Prompt Coding (GSPC)

To counter spatial forgetting, the GSPC module incorporates the average embedding of all patch tokens within a block into the prompts. This continuous "weaving" of global visual features ensures that prompts remain grounded in the image's actual structure throughout the network.

Long-term Prompt Coding (LPC)

Inspired by temporal coding in biological visual systems, the LPC module utilizes a weight-shared Long Short-Term Memory (LSTM) layer to manage prompts across blocks. Instead of a simple linear transfer from block $l$ to $l+1$, the LSTM serves as a memory bank, enabling the model to maintain and retrieve information from earlier stages of processing. This approach prevents the "token collapse" often seen in pure Transformer-based prompt aggregation.

LSPT adds only approximately 5% to 8% more parameters compared to the frozen backbone, making it significantly lighter than full fine-tuning.

Experimental Results

The framework was evaluated across 24 vision datasets, including the Fine-Grained Visual Classification (FGVC) suite, the VTAB-1K benchmark, and various medical imaging datasets.

Performance Benchmarks

LSPT consistently outperformed existing state-of-the-art methods such as VPT-Deep and GaPT:

  • FGVC Accuracy: Using a MAE pre-trained ViT-B, LSPT achieved an average accuracy of 76.94%, significantly higher than GaPT (73.39%) and VPT-Deep (72.02%).
  • VTAB-1K Structured Tasks: On tasks requiring geometric and structural understanding, LSPT improved the average score from 36.80% (GaPT) to 41.72%.
  • Generalization: The model demonstrated robust performance in specialized domains, including semantic segmentation (ADE-20K) and medical imaging (e.g., malaria detection, skin lesion classification).

Visualizing Spatial Awareness

The efficacy of LSPT is most evident in the visualization of attention maps. As shown in Figure 1, competing methods like GaPT exhibit significant noise in deeper layers (12th block). In contrast, LSPT maintains sharp focus on the target object, validating its ability to preserve spatial context through the network.

Comparison of attention maps between GaPT and LSPT across transformer blocks. LSPT maintains focused attention on the target object even in deeper layers (12th block), whereas GaPT's attention becomes noisy.

Discussion and Conclusion

Ablation studies indicate that the LSTM-based LPC is superior to additional Transformer layers for prompt management. The researchers hypothesize that the LSTM's "forget gate" is better suited for maintaining token variance and preventing the collapse of distinct information.

While LSPT is effective, future work may explore more sophisticated architectures for LPC and GSPC to further optimize the trade-off between computational cost and performance. Additionally, the concept of "long-term prompt memory" holds promise for multi-modal applications beyond pure computer vision.

Summary updated

Abstract

Visual Prompt Tuning (VPT) techniques have gained prominence for their capacity to adapt pre-trained Vision Transformers (ViTs) to downstream visual tasks using specialized learnable tokens termed as prompts. Contemporary VPT methodologies, especially when employed with self-supervised vision transformers, often default to the introduction of new learnable prompts or gated prompt tokens predominantly sourced from the model's previous block. A pivotal oversight in such approaches is their failure to harness the potential of long-range previous blocks as sources of prompts within each self-supervised ViT. To bridge this crucial gap, we introduce Long-term Spatial Prompt Tuning (LSPT) – a revolutionary approach to visual representation learning. Drawing inspiration from the intricacies of the human brain, LSPT ingeniously incorporates long-term gated prompts. This feature serves as temporal coding, curbing the risk of forgetting parameters acquired from earlier blocks. Further enhancing its prowess, LSPT brings into play patch tokens, serving as spatial coding. This is strategically designed to perpetually amass class-conscious features, thereby fortifying the model's prowess in distinguishing and identifying visual categories. To validate the efficacy of our proposed method, we engaged in rigorous experimentation across 5 FGVC and 19 VTAB-1K benchmarks. Our empirical findings underscore the superiority of LSPT, showcasing its ability to set new benchmarks in visual prompt tuning performance.
BibTeX
@misc{mo2024lspt,
  author = {Shentong Mo and Yansen Wang and Xufang Luo and Dongsheng Li},
  title = {{LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning}},
  year = {2024},
  eprint = {2402.17406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.17406},
}
Figure