Abstract
Token pruning has emerged as an effective solution to speed up the inference of large Transformer models. However, prior work on accelerating Vision Transformer (ViT) models requires training from scratch or fine-tuning with additional parameters, which prevents a simple plug-and-play. To avoid high training costs during the deployment stage, we present a fast training-free compression framework enabled by (i) a dense feature extractor in the initial layers; (ii) a sharpness-minimized model which is more compressible; and (iii) a local-global token merger that can exploit spatial relationships at various contexts. We applied our framework to various ViT and DeiT models and achieved up to 2x reduction in FLOPS and 1.8x speedup in inference throughput with <1% accuracy loss, while saving two orders of magnitude shorter training times than existing approaches.
Introduction
Vision Transformers (ViTs) have become the gold standard for many computer vision tasks, but their massive computational requirements make them difficult to deploy on edge devices or in real-time applications. To address this, researchers often utilize token reduction—pruning or merging redundant image "patches" to save on computation. However, most effective reduction techniques require expensive retraining or fine-tuning to maintain accuracy.
This paper introduces Delayed Spatial Merging (DSM), a training-free framework that accelerates ViT inference by up to 1.6x with negligible accuracy loss. By identifying a "delayed" onset of token similarity and utilizing hierarchical spatial merging, DSM outperforms existing methods while being two orders of magnitude faster to deploy since it requires zero retraining.
Methodology
The DSM framework improves upon vanilla token merging through two key strategies that respect the internal dynamics of the Transformer architecture:
1. Delayed Merging
By analyzing the cosine similarity of tokens across layers, the authors discovered a distinct "phase change." In the first few blocks of a ViT, tokens actually become less similar (divergent attention) as they extract unique features. Merging tokens during this phase is counterproductive and leads to significant accuracy drops. DSM waits until the tokens begin to converge—usually after the first 1/6th of the network—before initiating the merging process.
2. Hierarchical Spatial Merging
Instead of searching for similar tokens across the entire image immediately, DSM uses a window-based approach. It partitions the image into local windows and merges tokens within those windows first. As data flows deeper into the network, the window size progressively increases until it reaches global merging. This captures multi-scale redundancies more effectively than a one-size-fits-all global search and respects the inherent spatial structure of visual data.
Key Results and Insights
The researchers evaluated DSM across standard benchmarks including ImageNet-1k and various transfer learning tasks (e.g., CIFAR-100, Flowers102) using ViT and DeiT models ranging from Tiny to Huge scales.
- Efficiency and Speed: DSM achieves up to a 1.8× reduction in FLOPs and a 1.6× throughput speedup on an NVIDIA A6000 GPU. Because it is training-free, it can be applied to off-the-shelf models instantly, making it two orders of magnitude faster to implement than training-based competitors like E-ViT.
- Accuracy Preservation: On ImageNet-1k, DSM maintains accuracy within 1% of the baseline for ViT-S and ViT-B models, significantly outperforming vanilla Token Merging (ToMe).
- Semantic Coherence: Visualizations show that DSM merges contiguous, semantically similar patches (such as an entire animal limb), whereas previous methods often created fragmented, less logical token clusters.
Limitations and Future Work
While DSM reduces FLOPs significantly, the actual "wall-clock" speedup is slightly lower due to the overhead of data movement, such as sorting tokens and window partitioning. These effects are most noticeable in "Huge" models where the computational bottleneck shifts toward matrix multiplication. Future directions include optimizing CUDA kernels for window-stacking operations and exploring how delayed merging applies to other modalities like video or audio.
Summary updated