Faster Parameter-Efficient Tuning with Token Redundancy Reduction

Kwonyoung Kim, Jungin Park, Jin Kim, Hyeongjun Kwon, Kwanghoon Sohn

Abstract

Parameter-efficient tuning (PET) aims to transfer pre-trained foundation models to downstream tasks by learning a small number of parameters. Compared to traditional fine-tuning, which updates the entire model, PET significantly reduces storage and transfer costs for each task regardless of exponentially increasing pre-trained model capacity. However, most PET methods inherit the inference latency of their large backbone models and often introduce additional computational overhead due to additional modules (e.g. adapters), limiting their practicality for compute-intensive applications. In this paper, we propose Faster Parameter-Efficient Tuning (FPET), a novel approach that enhances inference speed and training efficiency while maintaining high storage efficiency. Specifically, we introduce a plug-and-play token redundancy reduction module delicately designed for PET. This module refines tokens from the self-attention layer using an adapter to learn the accurate similarity between tokens and cuts off the tokens through a fully-differentiable token merging strategy, which uses a straight-through estimator for optimal token reduction. Experimental results prove that our FPET achieves faster inference and higher memory efficiency than the pre-trained backbone while keeping competitive performance on par with state-of-the-art PET methods.

Introduction

Parameter-Efficient Tuning (PET) methods, such as LoRA and AdaptFormer, have become the standard for adapting large-scale foundation models to specific downstream tasks. By updating only a small fraction of the model's parameters, these methods drastically reduce storage and communication costs. However, a significant limitation remains: PET methods do not inherently improve the inference speed of the backbone model. In many cases, the addition of adapter modules actually increases computational overhead and memory consumption.

Existing research has primarily focused on storage efficiency, leaving the computational bottleneck of large backbone models like Vision Transformers (ViTs) unaddressed. This paper introduces Faster Parameter-Efficient Tuning (FPET), a method designed to optimize both storage and computational efficiency by integrating a token redundancy reduction module directly into the tuning process.

Methodology: Differentiable Token Merging

The core of FPET is a "plug-and-play" token redundancy reduction module that leverages an evolved version of Token Merging (ToMe). Unlike previous heuristic-based approaches that are non-differentiable, FPET makes the merging process fully differentiable using a Straight-Through Estimator (STE). This allows the model to learn task-specific token importance during training.

Comparison of (a) traditional bipartite soft matching and (b) the proposed bipartite differentiable matching in FPET.

The FPET module follows several key steps:

  1. Key Refinement: It extracts and refines "keys" from the self-attention layer using a lightweight adapter. This ensures that the similarity calculation used for merging is optimized for the specific task.
  2. Checkerboard Partitioning: Tokens are assigned to two sets in a checkerboard pattern to preserve spatial context and prevent the formation of unnatural artifacts during merging.
  3. Differentiable Matching: Similarities are calculated between the two sets, and a matching matrix is formed. Through matrix multiplication and the STE, the model identifies and merges redundant tokens while allowing gradients to flow back to the refinement adapter.
  4. Strategic Placement: To balance information preservation and speed, FPET performs a single significant reduction (e.g., merging 50% of tokens) at the middle layer of the backbone (e.g., layer 6 of a 12-layer ViT).
  5. Gradient Management: While gradients are necessary to train the merging adapter, they are not propagated back into the frozen backbone, preventing interference with pre-trained representations.

Experimental Results

The authors evaluated FPET on several benchmarks, including VTAB-1K, FGVC, and Flickr30K. The results demonstrate that FPET achieves state-of-the-art accuracy while being significantly faster and more memory-efficient than existing PET methods.

Performance comparison of FPET (Ours) against other PET methods in terms of (a) Accuracy vs. Inference Time and (b) Accuracy vs. GPU Memory usage.

Key findings include:

  • Inference Speed: FPET achieved a 19.8% increase in inference speed compared to the original pre-trained backbone, effectively overcoming the "latency paradox" of standard PET.
  • Memory Efficiency: GPU memory usage was reduced by approximately 40% compared to full fine-tuning, and it remained lower than other popular PET methods like LoRA or AdaptFormer.
  • Accuracy: On the VTAB-1K benchmark, FPET (integrated with Bi-AdaptFormer) achieved 77.0% accuracy, matching the performance of much slower state-of-the-art methods.
  • FLOPs Reduction: The method successfully reduced total floating-point operations (FLOPs) by an average of 24%.

Discussion and Conclusion

FPET demonstrates that it is possible to make large foundation models faster than their original state during task adaptation. By making token merging differentiable and task-aware, the model learns to prune redundant information that is not critical for the target task.

While currently optimized for Vision Transformers, future work could extend these principles to other architectures like Swin Transformers or dense prediction tasks like object detection. FPET represents a significant step toward making large-scale model adaptation practical for compute-constrained environments without sacrificing performance.

Summary updated

Abstract

Parameter-efficient tuning (PET) aims to transfer pre-trained foundation models to downstream tasks by learning a small number of parameters. Compared to traditional fine-tuning, which updates the entire model, PET significantly reduces storage and transfer costs for each task regardless of exponentially increasing pre-trained model capacity. However, most PET methods inherit the inference latency of their large backbone models and often introduce additional computational overhead due to additional modules (e.g. adapters), limiting their practicality for compute-intensive applications. In this paper, we propose Faster Parameter-Efficient Tuning (FPET), a novel approach that enhances inference speed and training efficiency while maintaining high storage efficiency. Specifically, we introduce a plug-and-play token redundancy reduction module delicately designed for PET. This module refines tokens from the self-attention layer using an adapter to learn the accurate similarity between tokens and cuts off the tokens through a fully-differentiable token merging strategy, which uses a straight-through estimator for optimal token reduction. Experimental results prove that our FPET achieves faster inference and higher memory efficiency than the pre-trained backbone while keeping competitive performance on par with state-of-the-art PET methods.
BibTeX
@inproceedings{kim2025faster,
  author = {Kwonyoung Kim and Jungin Park and Jin Kim and Hyeongjun Kwon and Kwanghoon Sohn},
  title = {{Faster Parameter-Efficient Tuning with Token Redundancy Reduction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=7tnSusJ6j3},
}
Figure