Rethinking Token Reduction with Parameter-Efficient Fine-Tuning in ViT for Pixel-Level Tasks

Cheng Lei, Ao Li, Hu Yao, Ce Zhu, Le ZhangCVPR 2025 · DOI

Abstract

While Parameter-Efficient Fine-Tuning (PEFT) significantly reduces the training costs associated with Vision Transformers (ViTs), inference remains computationally expensive due to the high number of tokens required for high-resolution images. This is particularly problematic for pixel-level tasks such as segmentation, where standard token reduction (TR) methods often degrade spatial detail and attention diversity. This paper introduces the Diverse Attention Restorer (DAR), a novel framework that decouples token reduction from task-specific adaptation. By utilizing a dedicated token compensator and strategic post-FFN masking, DAR maintains the fine-grained spatial information and attention richness necessary for high-precision computer vision while significantly reducing computational overhead.

Introduction

Vision Transformers (ViTs) have set new benchmarks in computer vision, but their quadratic complexity relative to sequence length makes them difficult to deploy in resource-constrained environments. While PEFT techniques like LoRA or Adapters allow for efficient tuning of pre-trained models, they do not address the inference-time bottleneck caused by long token sequences. In image classification, "unimportant" tokens can be dropped with minimal impact. However, for pixel-level tasks—including medical imaging, camouflaged object detection, and salient object segmentation—maintaining a dense spatial grid is crucial for accurate mask generation.

Existing token reduction methods often fail in these scenarios due to two primary factors: Attention Homogenization, where different attention heads begin to focus on identical features, and Bias Perturbation, where standard masking interferes with the pre-trained biases of the model's Feed-Forward Networks (FFN).

The DAR-TR-PEFT Framework

To address these challenges, the authors propose DAR-TR-PEFT, a decoupled architecture designed to preserve model expressivity during token reduction. The framework consists of several key innovations:

  • Decoupled Token Compensator (TC): Rather than relying on a single adapter to handle both task adaptation and spatial restoration, DAR introduces a lightweight TC. This component uses depth-wise convolutions to inject local spatial biases, effectively restoring the high-frequency information (edges and textures) that global self-attention might lose when tokens are sparse.
  • Strategic Post-FFN Masking: The researchers discovered that applying token masks before the FFN causes internal bias weights to "pollute" masked regions, hindering convergence. By shifting the learnable mask to occur after the FFN, the model preserves the integrity of pre-trained biases.
  • Learnable Masking with Gumbel-Softmax: The system employs a 3x3 convolution to generate dynamic, input-dependent masks, allowing the model to intelligently select which tokens are most relevant for a specific image.
  • Feature Distillation: To prevent the reduced-token model from drifting away from the original's rich feature representations, a distillation loss is used to align the fine-tuned features with the pre-trained foundation model.

Empirical Analysis: Restoring Attention Diversity

A critical contribution of this work is the quantitative analysis of how token reduction affects internal model dynamics. The authors utilize Mean Attention Distance (MAD) and Normalized Mutual Information (NMI) to measure attention diversity.

Comprehensive analysis of attention diversity and feature quality. Subplots (a-c) show Mean Attention Distance (MAD), (d) shows Normalized Mutual Information (NMI), and (e-f) show Relative Log Amplitude, comparing the baseline, standard Token Reduction (TR), and the proposed DAR method.

As shown in the analysis above, standard token reduction (TR) leads to a collapse in MAD and NMI, indicating that attention heads become redundant. The DAR method successfully restores this diversity, maintaining a high MAD across blocks and ensuring that different attention heads continue to capture varied aspects of the scene. Furthermore, the log amplitude metrics demonstrate that DAR preserves the feature richness of the original DINOv2 backbone more effectively than standard TR approaches.

Experimental Evaluation

The DAR-TR-PEFT method was evaluated across multiple domains to demonstrate its robustness and generalization capabilities:

  • Salient Object Segmentation (DUTS): DAR achieved a weighted F-measure ($F_\beta^w$) of 0.901 using only 91.54 GFLOPs, outperforming the DyT baseline (0.859 at 95.51 GFLOPs).
  • Medical Imaging (Kvasir): In polyp segmentation, DAR achieved a score of 0.940, nearly matching full-token models while offering significantly faster inference.
  • Camouflaged Object Detection (COD10K): The method proved highly effective in complex scenes where fine-grained detail is paramount.
  • Zero-Shot Generalization: When trained on simple saliency tasks and tested on camouflaged objects, DAR outperformed specialized zero-shot models like GenSAM, highlighting its ability to learn fundamental spatial relationships.

Conclusion

The DAR-TR-PEFT framework represents a significant step forward in making high-performance ViTs practical for pixel-level tasks. By identifying and solving the issues of attention homogenization and bias perturbation, the authors demonstrate that it is possible to achieve substantial token reduction without sacrificing the spatial precision required for advanced segmentation. Future work may explore extending this decoupled design to video-based tasks and other transformer architectures.

Summary updated

BibTeX
@inproceedings{lei2025rethinking,
  author = {Cheng Lei and Ao Li and Hu Yao and Ce Zhu and Le Zhang},
  title = {{Rethinking Token Reduction with Parameter-Efficient Fine-Tuning in ViT for Pixel-Level Tasks}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01393},
}
Figure