TL;DR
SWAM (Saliency-Weighted Asymmetric Merging) is a new framework that makes Vision Transformers (ViTs) faster by intelligently merging redundant image patches. By using a lightweight "saliency" predictor to identify and protect important tokens, SWAM achieves up to a 1.46x speedup with minimal accuracy loss—and in some cases, even improves performance.
What Problem Does This Paper Address?
Vision Transformers have revolutionized computer vision, but they come with a steep computational price. The core mechanism, self-attention, has a "quadratic complexity" problem: if you double the number of image patches (tokens), the computation required quadruples.
To solve this, researchers use token reduction—merging or pruning patches as they move through the network's layers. A popular method is Bipartite Soft Matching (BSM), which merges tokens based on how similar their features are. However, the authors identify two major flaws in this "similarity-only" approach:
- The Importance Blindspot: Two patches might look very similar (high similarity) but be critical for identifying the object (high saliency). Similarity-only methods might merge these important patches, losing fine-grained detail.
- The Dilution Effect: When two tokens are merged, they are usually averaged. This can "water down" the information from a highly relevant patch if it's merged with a less relevant one.
Methods
SWAM introduces a training-based framework built on top of the DeiT-Small architecture. It adds three lightweight saliency prediction heads (MLPs) that account for only about 1% of the total model parameters.
The framework operates in three main steps at each reduction stage:
- Saliency Scoring: The MLP predicts an importance score for every token. To train this, the authors use "Attention Rollout"—a technique that tracks how information flows from the class (CLS) token back to the image patches—as a supervisory signal.
- Multiplicative Max-Saliency Pair Scoring: Instead of just looking at cosine similarity to decide which tokens to merge, SWAM applies a penalty. If even one token in a potential pair has a high saliency score, the pair is penalized, making it much less likely to be merged.
- Asymmetric Merging: When a merge does happen, SWAM doesn't just average the tokens. It uses an exponential weighting scheme based on saliency. The more "important" token contributes significantly more to the resulting merged token, preserving its semantic content.
Datasets
The authors evaluated SWAM on three standard image recognition benchmarks:
- CIFAR-100: 50,000 training images across 100 categories.
- TinyImageNet: 200 categories with 100,000 training images.
- ImageNet-100: A 100-class subset of the massive ImageNet-1K dataset, used here for in-domain fine-tuning.
Key Insights and Ideas
- Protecting the "Stars": The use of a multiplicative maximum saliency penalty is a clever "fail-safe." By focusing on the most important token in a pair rather than the average importance, the model ensures that high-value information is rarely compromised.
- Asymmetry as a Feature: Traditional merging treats all tokens as equal partners. SWAM recognizes that in a neural network, some features are simply more "correct" or useful for the task than others. Letting the salient token "win" the merge prevents information decay.
- Learned Efficiency: Unlike "plug-and-play" methods that work on frozen models, SWAM is trained end-to-end. This allows the saliency heads to become task-adaptive, learning exactly what features the specific classification task needs to keep.
Key Results
The researchers tested two configurations: Conservative (reducing 196 tokens down to 100) and Aggressive (reducing down to 70).
- Speed and Efficiency: The Conservative schedule achieved a 27.5% FLOPs reduction and a 1.35x speedup. The Aggressive schedule pushed this to a 33.8% FLOPs reduction and a 1.46x speedup.
- Accuracy Gains: Surprisingly, on ImageNet-100, SWAM actually improved accuracy over the full-token baseline by +0.26% (Conservative) and +0.23% (Aggressive).
- Ablation Success: In direct comparisons on CIFAR-100, SWAM's multiplicative max-saliency approach (87.69% top-1) outperformed both standard BSM/ToMe (87.25%) and a simpler additive saliency method (87.50%).
Limitations and Future Work
While the results are promising, the authors note a few areas for further exploration:
- Model Scaling: All experiments were conducted using DeiT-Small. It remains to be seen how well these saliency mechanisms generalize to much smaller (Tiny) or much larger (Base/Large) ViT backbones.
- Dataset Scope: The current study focused on subsets like ImageNet-100. Testing on the full ImageNet-1K dataset is a logical next step to prove industrial-scale viability.
- Hardware Constraints: Due to quota limits on the Kaggle platform used for training, the team couldn't explore even more complex per-block reduction strategies.
Summary updated