Abstract
Vision Transformers (ViTs) have established themselves as a cornerstone of modern computer vision, yet their computational demands—driven by the quadratic complexity of self-attention—remain a significant barrier to deployment. Token compression methods, such as pruning and merging, have emerged as effective strategies to reduce this overhead. However, existing techniques often rely on heuristic-based, uniform compression rates across all layers, failing to account for the varying redundancy present in different stages of the network. This paper introduces DiffRate, a novel framework that treats the compression rate as a learnable parameter. By employing a Differentiable Discrete Proxy (DDP), DiffRate enables the optimization of layer-wise compression schedules through standard backpropagation. Experimental results on ImageNet-1k demonstrate that DiffRate can reduce FLOPs by 40% with a minimal accuracy loss of 0.16%, outperforming existing state-of-the-art methods while remaining compatible with off-the-shelf pre-trained models.
1. Introduction
The success of Vision Transformers (ViTs) is largely attributed to their ability to model long-range dependencies through self-attention. However, this power comes at the cost of $O(N^2)$ complexity, where $N$ is the number of input tokens (image patches). As resolution increases or models scale to "Huge" variants, the computational burden becomes prohibitive.
To address this, researchers have developed token compression techniques. Pruning involves identifying and discarding "unimportant" tokens (e.g., background patches), while merging combines similar tokens to preserve information while reducing the total count. Despite their effectiveness, a critical bottleneck remains: the "compression schedule." Determining how many tokens to keep at each layer is currently a manual "guess-and-check" process. Since a typical ViT may have 12 to 24 layers, the search space for an optimal schedule is vast, and uniform rates are rarely optimal.
2. The DiffRate Framework
DiffRate addresses the limitations of manual tuning by making the compression rate differentiable. The core of the method is the Differentiable Discrete Proxy (DDP) module, which bridges the gap between discrete token counts and continuous gradient-based optimization.
2.1 Unified Pruning and Merging
Unlike previous methods that focus on either pruning or merging, DiffRate integrates both into a single framework. The process begins by ranking tokens based on their "class attention"—the attention weights assigned to the [CLS] token, which serves as a proxy for a patch's contribution to the final classification. The module then prunes the least significant tokens and merges the remaining low-importance tokens with high-importance ones, ensuring that foreground information is condensed rather than lost.
2.2 Differentiable Optimization
To optimize the number of tokens $K$ per layer, DiffRate re-parameterizes the selection process. Instead of choosing a hard integer, the model learns a probability distribution over a range of candidate rates. During the "search" phase, the model uses an attention mask to simulate the removal of tokens without actually deleting them, allowing gradients to flow through the compression parameters.
The optimization objective is defined by a composite loss function that combines standard classification loss with a penalty term that encourages the model to stay within a specific computational (FLOPs) budget.
3. Experimental Evaluation
The authors evaluated DiffRate primarily on the ImageNet-1k dataset across various model scales, including ViT-Small, Base, Large, and Huge, as well as downstream tasks on ADE20K.
3.1 Efficiency and Accuracy
DiffRate demonstrates remarkable efficiency gains. For a ViT-Huge model pre-trained with Masked Autoencoders (MAE), DiffRate achieved a 40% reduction in FLOPs and a 1.5× increase in throughput with only a 0.16% drop in Top-1 accuracy. Notably, these results were achieved "off-the-shelf," meaning the learned compression schedules were applied to the frozen weights of the pre-trained model without further fine-tuning.
3.2 Comparison with State-of-the-Art
When compared to leading methods such as ToMe (Token Merging) and EViT (Evolutionary ViT), DiffRate consistently provides a superior trade-off between accuracy and speed. The automated search process is also highly efficient, requiring only about 2.7 GPU hours for a ViT-Base model—using just 1,000 images over 3 epochs.
3.3 Hardware-Aware Compression
Beyond theoretical FLOPs, DiffRate can be tailored to specific hardware constraints. By replacing the FLOPs penalty in the loss function with hardware-specific metrics like latency or power consumption, the framework can optimize models for specific edge devices or server environments.
4. Discussion and Insights
A key insight from this work is the synergy between pruning and merging. Pruning is highly effective at removing background noise, while merging excels at condensing redundant foreground features. By allowing the model to learn the balance between these two operations per layer, DiffRate achieves higher fidelity than fixed-rate approaches.
However, challenges remain. Applying DiffRate to hierarchical architectures like Swin Transformers is less straightforward due to the shifting spatial resolutions of those models. While the authors propose an "uncompression" module to bridge this gap, it introduces additional complexity. Future research may focus on "zero-shot" compression estimation to eliminate the need for the initial search phase entirely.
5. Conclusion
DiffRate represents a significant step toward automated model efficiency. By making the compression rate differentiable, it removes the need for manual hyperparameter tuning and provides a practical, high-performance solution for deploying large-scale Vision Transformers in resource-constrained environments.
Summary updated