Abstract
As the scale of vision models continues to grow, Visual Prompt Tuning (VPT) has emerged as a parameter-efficient transfer learning technique, noted for its superior performance compared to full fine-tuning. However, indiscriminately applying prompts to every layer without considering their inherent correlations, can cause significant disturbances, leading to suboptimal transferability. Additionally, VPT disrupts the original self-attention structure, affecting the aggregation of visual features, and lacks a mechanism for explicitly mining discriminative visual features, which are crucial for classification. To address these issues, we propose a Semantic Hierarchical Prompt (SHIP) fine-tuning strategy. We adaptively construct semantic hierarchies and use semantic-independent and semantic-shared prompts to learn hierarchical representations. We also integrate attribute prompts and a prompt matching loss to enhance feature discrimination and employ decoupled attention for robustness and reduced inference costs. SHIP significantly improves performance, achieving a 4.9% gain in accuracy over VPT with a ViT-B/16 backbone on VTAB-1k tasks. Our code is available at https://github.com/haoweiz23/SHIP.
Introduction
As Vision Transformers (ViTs) scale, fine-tuning all parameters for new tasks becomes computationally prohibitive. Visual Prompt Tuning (VPT) addresses this by inserting learnable prompt tokens into the input sequence while keeping the backbone frozen. However, standard VPT has three major flaws: it applies prompts uniformly across layers regardless of their semantic abstraction level, it disrupts the original self-attention mechanism, and it lacks a mechanism to capture discriminative visual features explicitly.
Analysis of inter-layer affinity (Figure 1) shows that ViT layers do not evolve linearly but rather in "semantic plateaus" followed by transitions. This observation motivates the Semantic Hierarchical Prompt (SHIP) strategy, which groups layers into hierarchies to better align prompts with the model's internal representations.
Methodology
The SHIP framework introduces several novel components to make prompting more aware of the model's structure and the task's requirements.
Semantic Hierarchical Partitioning
SHIP calculates the cosine similarity (affinity) between features of adjacent layers and uses a greedy search to group similar layers into semantic hierarchies. This partitioning is task-adaptive; for instance, object-level tasks show transitions in later layers, while texture-based tasks transition earlier.
Hierarchical Prompts: SIP and SSP
SHIP employs two types of prompts within its hierarchies:
- Semantic-Independent Prompts (SIP): Unique to each hierarchy to capture specific levels of abstraction.
- Semantic-Shared Prompts (SSP): Applied across all layers to maintain a consistent global representation.
Attribute Prompts and Decoupled Attention
To capture discriminative features like textures or shapes, SHIP uses Attribute Prompts (AP) derived from clustering training data. A Prompt Matching Loss (PML) aligns these prompts with the most important image tokens. Furthermore, Decoupled Attention (DA) reformulates the attention mechanism to preserve the original image feature aggregation, preventing "attention dilution" caused by the added prompt tokens.
Experimental Results
Evaluation on the VTAB-1k benchmark (19 tasks across Natural, Specialized, and Structured categories) demonstrates SHIP's effectiveness.
- Superior Accuracy: SHIP achieved an average Top-1 accuracy of 76.9%, outperforming VPT (72.0%) and other advanced methods like NOAH (75.5%).
- Parameter Efficiency: These gains are achieved with only 0.38M trainable parameters, making it more efficient than many competing prompt-based methods.
- Overfitting Mitigation: SHIP's hierarchical approach acts as a regularizer. While standard VPT often begins to overfit around epoch 20, SHIP’s test loss continues to decrease steadily, indicating superior generalization.
Conclusion
SHIP represents a significant advancement in parameter-efficient fine-tuning by moving away from uniform prompting. By respecting the semantic hierarchies of vision models and explicitly mining discriminative features, SHIP achieves state-of-the-art performance with minimal overhead. Future work may extend this hierarchical prompting logic to other modalities, such as large language models.
Summary updated