Abstract
Parameter-efficient tuning (PET) aims to transfer pre-trained foundation models to downstream tasks by learning a small number of parameters. Compared to traditional fine-tuning, which updates the entire model, PET significantly reduces storage and transfer costs. However, most PET methods inherit the inference latency of their large backbone models and often introduce additional computational overhead due to additional modules (e.g., adapters), limiting their practicality for compute-intensive applications. We propose Faster Parameter-Efficient Tuning (FPET), a novel approach that enhances inference speed and training efficiency while maintaining high storage efficiency. Specifically, we introduce a plug-and-play token redundancy reduction module delicately designed for PET. This module refines tokens from the self-attention layer using an adapter to learn the accurate similarity between tokens and cuts off the tokens through a fully-differentiable token merging strategy, which uses a straight-through estimator for optimal token reduction. Experimental results prove that our FPET achieves faster inference and higher memory efficiency than the pre-trained backbone while keeping competitive performance on par with state-of-the-art PET methods.
Introduction
The rise of massive pre-trained foundation models has made Parameter-Efficient Tuning (PET) the standard for task adaptation. By training only a tiny fraction of parameters (e.g., via LoRA or Adapters), researchers can save significant storage. However, a critical bottleneck remains: inference latency. Even with minimal parameter updates, the entire backbone model must be executed. In many cases, the addition of PET modules actually increases latency. Existing token reduction techniques, which aim to skip redundant image patches, often lack differentiability, preventing them from being optimized end-to-end during the PET process.
The FPET Framework
FPET addresses these challenges by integrating a "plug-and-play" token redundancy reduction module into the PET pipeline. The core innovation lies in making the token selection process learnable and task-specific.
Bipartite Differentiable Matching
Unlike previous heuristic-based methods, FPET utilizes a Straight-Through Estimator (STE) to make the discrete "pick and merge" process differentiable. This allows the model to learn which tokens are redundant for a specific downstream task during training.
Key Refinement and Spatial Priors
To ensure accurate matching, FPET uses a lightweight adapter to refine the "keys" from the self-attention layer. This ensures that redundancy is identified based on features relevant to the target task. Furthermore, a checkerboard spatial prior is employed to split tokens into sets, ensuring that merging occurs between spatially adjacent neighbors, thus preserving the structural integrity of the image features.
Strategic Merging
To balance speed and accuracy, FPET performs a one-time "halving" of tokens at the middle layer of the Vision Transformer (e.g., the 6th layer of a 12-layer ViT). This avoids the overhead of per-layer merging while providing substantial computational savings. The authors found that halting gradients at the backbone interface prevented interference between pre-trained features and the new merging logic.
Experimental Results
The framework was evaluated on the Visual Task Adaptation Benchmark (VTAB-1K), which includes 19 diverse tasks across natural, specialized, and structured domains.
Efficiency and Performance
FPET demonstrates significant improvements across multiple efficiency metrics compared to state-of-the-art PET methods like LoRA, AdaptFormer, and Bi-LoRA:
- Inference Speed: Achieved a ~19.8% increase in inference speed over the original backbone.
- Memory Usage: Reduced GPU memory consumption during training by approximately 40%.
- Computational Cost: Reduced total FLOPs by roughly 24%.
As shown in the figures above, FPET (represented by stars) consistently occupies the top-left regions of the charts, indicating superior accuracy-to-efficiency trade-offs. For instance, using the Bi-AdaptFormer backbone, FPET reached 77.0% accuracy on VTAB-1K while being significantly faster and more memory-efficient than the baseline.
Conclusion
FPET successfully bridges the gap between storage efficiency and computational speed in parameter-efficient tuning. By introducing a differentiable token-merging strategy, it allows foundation models to be adapted to new tasks with lower latency and memory requirements than the original pre-trained backbones. Future work will explore dynamic reduction rates and optimizations for hierarchical transformer architectures.
Summary updated