SWAM: Saliency-Weighted Asymmetric Merging for Efficient Vision Transformers

Utkarsh Kumar Singh, Manish Das, Shiv Ram Dubey, Satish Kumar SingharXiv 2026 · arXiv

TL;DR

SWAM (Saliency-Weighted Asymmetric Merging) is a new framework that makes Vision Transformers (ViTs) faster by intelligently merging redundant image patches. By using a lightweight "saliency" predictor to identify and protect important tokens, SWAM achieves up to a 1.46x speedup with minimal accuracy loss—and in some cases, even improves performance.

What Problem Does This Paper Address?

Vision Transformers have revolutionized computer vision, but they come with a steep computational price. The core mechanism, self-attention, has a "quadratic complexity" problem: if you double the number of image patches (tokens), the computation required quadruples.

To solve this, researchers use token reduction—merging or pruning patches as they move through the network's layers. A popular method is Bipartite Soft Matching (BSM), which merges tokens based on how similar their features are. However, the authors identify two major flaws in this "similarity-only" approach:

  1. The Importance Blindspot: Two patches might look very similar (high similarity) but be critical for identifying the object (high saliency). Similarity-only methods might merge these important patches, losing fine-grained detail.
  2. The Dilution Effect: When two tokens are merged, they are usually averaged. This can "water down" the information from a highly relevant patch if it's merged with a less relevant one.

Methods

SWAM introduces a training-based framework built on top of the DeiT-Small architecture. It adds three lightweight saliency prediction heads (MLPs) that account for only about 1% of the total model parameters.

The framework operates in three main steps at each reduction stage:

  1. Saliency Scoring: The MLP predicts an importance score for every token. To train this, the authors use "Attention Rollout"—a technique that tracks how information flows from the class (CLS) token back to the image patches—as a supervisory signal.
  2. Multiplicative Max-Saliency Pair Scoring: Instead of just looking at cosine similarity to decide which tokens to merge, SWAM applies a penalty. If even one token in a potential pair has a high saliency score, the pair is penalized, making it much less likely to be merged.
  3. Asymmetric Merging: When a merge does happen, SWAM doesn't just average the tokens. It uses an exponential weighting scheme based on saliency. The more "important" token contributes significantly more to the resulting merged token, preserving its semantic content.
Fig. 1. SWAM pipeline. Top: DeiT-Small backbone with three SWAM reduction stages inserted after blocks 4, 7, and 10. Patch-token counts decrease from 196 to 100 (conservative, C) or 70 (aggressive, A) while the CLS token is retained throughout all reduction stages. Bottom: Detail of a single stage. The saliency head predicts per-token importance scores that feed into multiplicative max-saliency pair scoring (Eq. 6) and asymmetric saliency-weighted merging (Eq. 9).

Datasets

The authors evaluated SWAM on three standard image recognition benchmarks:

  • CIFAR-100: 50,000 training images across 100 categories.
  • TinyImageNet: 200 categories with 100,000 training images.
  • ImageNet-100: A 100-class subset of the massive ImageNet-1K dataset, used here for in-domain fine-tuning.

Key Insights and Ideas

  • Protecting the "Stars": The use of a multiplicative maximum saliency penalty is a clever "fail-safe." By focusing on the most important token in a pair rather than the average importance, the model ensures that high-value information is rarely compromised.
  • Asymmetry as a Feature: Traditional merging treats all tokens as equal partners. SWAM recognizes that in a neural network, some features are simply more "correct" or useful for the task than others. Letting the salient token "win" the merge prevents information decay.
  • Learned Efficiency: Unlike "plug-and-play" methods that work on frozen models, SWAM is trained end-to-end. This allows the saliency heads to become task-adaptive, learning exactly what features the specific classification task needs to keep.

Key Results

The researchers tested two configurations: Conservative (reducing 196 tokens down to 100) and Aggressive (reducing down to 70).

  • Speed and Efficiency: The Conservative schedule achieved a 27.5% FLOPs reduction and a 1.35x speedup. The Aggressive schedule pushed this to a 33.8% FLOPs reduction and a 1.46x speedup.
  • Accuracy Gains: Surprisingly, on ImageNet-100, SWAM actually improved accuracy over the full-token baseline by +0.26% (Conservative) and +0.23% (Aggressive).
  • Ablation Success: In direct comparisons on CIFAR-100, SWAM's multiplicative max-saliency approach (87.69% top-1) outperformed both standard BSM/ToMe (87.25%) and a simpler additive saliency method (87.50%).
Table showing per-seed accuracy results and speedups across different datasets and schedules.

Limitations and Future Work

While the results are promising, the authors note a few areas for further exploration:

  • Model Scaling: All experiments were conducted using DeiT-Small. It remains to be seen how well these saliency mechanisms generalize to much smaller (Tiny) or much larger (Base/Large) ViT backbones.
  • Dataset Scope: The current study focused on subsets like ImageNet-100. Testing on the full ImageNet-1K dataset is a logical next step to prove industrial-scale viability.
  • Hardware Constraints: Due to quota limits on the Kaggle platform used for training, the team couldn't explore even more complex per-block reduction strategies.

Summary updated

Abstract

Vision Transformers (ViTs) achieve strong performance on image recognition tasks, yet their quadratic self-attention complexity over input tokens creates prohibitive computational costs for practical deployment. Token reduction methods offer a practical solution by progressively reducing the token sequence during the forward pass. We propose Saliency-Weighted Asymmetric Merging (SWAM), a training-based token merging framework that extends the Bipartite Soft Matching (BSM) with two saliency-aware mechanisms. We introduce a two-step matching formulation in which cosine similarity decides candidate pairs, while a multiplicative maximum saliency penalty regulates their selection for merging. We then further use asymmetric exponential saliency-weighted aggregation, which allows more salient tokens to contribute more strongly to the merged representation. A lightweight MLP-based saliency prediction module (75k parameters per stage, 1% of backbone parameters) supervised via attention rollout is trained end-to-end jointly with a DeiT-Small backbone using a composite cross-entropy, knowledge-distillation, and saliency-alignment objective. On CIFAR-100, TinyImageNet, and ImageNet-100, under a conservative schedule that reduces tokens from 196 to 100 patch tokens, with the CLS token retained throughout, 27.5% FLOPs reduction and 1.35 times speedup, SWAM matches or exceeds baseline across the reported results. Under an aggressive schedule (196 to 70 tokens with the CLS token kept throughout, 33.8% FLOPs reduction, and 1.46 times speedup), accuracy gains are achieved on ImageNet-100 (+0.23%) with minimal losses on CIFAR-100 and TinyImageNet.
BibTeX
@misc{singh2026swam,
  author = {Utkarsh Kumar Singh and Manish Das and Shiv Ram Dubey and Satish Kumar Singh},
  title = {{SWAM: Saliency-Weighted Asymmetric Merging for Efficient Vision Transformers}},
  year = {2026},
  eprint = {2611.18057},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18057},
}
Figure