Introduction
Adapting large-scale, self-supervised Vision Transformers (ViTs) to specific downstream tasks is a computationally intensive process. While Visual Prompt Tuning (VPT) offers a parameter-efficient alternative by training only a small set of prompt tokens while keeping the backbone frozen, it suffers from two significant "forgetting" phenomena:
- Temporal Forgetting: Information from early transformer blocks decays exponentially as it propagates through the network, leading to a loss of long-term context in deeper layers.
- Spatial Forgetting: Prompts frequently lose their structural connection to image patch tokens, resulting in degraded spatial awareness and "blurry" attention maps.
LSPT (Long-term Spatial Prompt Tuning) addresses these issues by integrating a long-term memory mechanism and global spatial coding, setting new performance benchmarks across diverse vision datasets while remaining highly parameter-efficient.
Methodology
LSPT introduces two core modules designed to mitigate information loss and maintain structural grounding:
Global Spatial Prompt Coding (GSPC)
To counter spatial forgetting, the GSPC module incorporates the average embedding of all patch tokens within a block into the prompts. This continuous "weaving" of global visual features ensures that prompts remain grounded in the image's actual structure throughout the network.
Long-term Prompt Coding (LPC)
Inspired by temporal coding in biological visual systems, the LPC module utilizes a weight-shared Long Short-Term Memory (LSTM) layer to manage prompts across blocks. Instead of a simple linear transfer from block $l$ to $l+1$, the LSTM serves as a memory bank, enabling the model to maintain and retrieve information from earlier stages of processing. This approach prevents the "token collapse" often seen in pure Transformer-based prompt aggregation.
LSPT adds only approximately 5% to 8% more parameters compared to the frozen backbone, making it significantly lighter than full fine-tuning.
Experimental Results
The framework was evaluated across 24 vision datasets, including the Fine-Grained Visual Classification (FGVC) suite, the VTAB-1K benchmark, and various medical imaging datasets.
Performance Benchmarks
LSPT consistently outperformed existing state-of-the-art methods such as VPT-Deep and GaPT:
- FGVC Accuracy: Using a MAE pre-trained ViT-B, LSPT achieved an average accuracy of 76.94%, significantly higher than GaPT (73.39%) and VPT-Deep (72.02%).
- VTAB-1K Structured Tasks: On tasks requiring geometric and structural understanding, LSPT improved the average score from 36.80% (GaPT) to 41.72%.
- Generalization: The model demonstrated robust performance in specialized domains, including semantic segmentation (ADE-20K) and medical imaging (e.g., malaria detection, skin lesion classification).
Visualizing Spatial Awareness
The efficacy of LSPT is most evident in the visualization of attention maps. As shown in Figure 1, competing methods like GaPT exhibit significant noise in deeper layers (12th block). In contrast, LSPT maintains sharp focus on the target object, validating its ability to preserve spatial context through the network.
Discussion and Conclusion
Ablation studies indicate that the LSTM-based LPC is superior to additional Transformer layers for prompt management. The researchers hypothesize that the LSTM's "forget gate" is better suited for maintaining token variance and preventing the collapse of distinct information.
While LSPT is effective, future work may explore more sophisticated architectures for LPC and GSPC to further optimize the trade-off between computational cost and performance. Additionally, the concept of "long-term prompt memory" holds promise for multi-modal applications beyond pure computer vision.
Summary updated