Similarity-Aware Token Pruning: Your VLM but Faster

Ahmadreza Jeddi, Negin Baghbanzadeh, Elham Dolatabadi, Babak Taati

Introduction

As Vision Transformers (ViTs) and Vision-Language Models (VLMs) continue to scale in both resolution and parameter count, they encounter a significant computational bottleneck: the quadratic complexity of self-attention. Because every token must attend to every other token, increasing input size leads to an exponential rise in processing requirements. While "token pruning"—the removal of redundant image patches—offers a potential solution, existing methods often require expensive retraining or rely on static pruning rates that ignore the internal dynamics of the model.

This paper introduces SAINT (Similarity-Aware INference Time Token Pruning), a training-free, plug-and-play framework designed to accelerate inference by identifying and removing redundant tokens based on their feature similarity. SAINT achieves significant speedups, such as doubling the throughput of ViT-H or reducing LLaVA-13B’s visual token count by 75%, with negligible impact on accuracy.

The SAINT Framework

SAINT treats token pruning as a graph sparsification problem. Unlike traditional methods that use noisy attention maps to determine token importance, SAINT leverages the inherent redundancy in token representations. The methodology follows a four-step process:

  1. Bipartite Graph Construction: To avoid the $O(N^2)$ cost of full similarity comparisons, SAINT partitions tokens into two sets and constructs a bipartite graph. Edges are weighted by the cosine similarity of the tokens' "keys."
  2. Adaptive Pruning via Voting: Rather than dropping a fixed percentage of tokens, SAINT employs a dynamic "voting" mechanism. Tokens with a high degree of similarity to many neighbors are flagged as redundant. This allows the model to adapt the pruning rate per batch based on actual image content.
  3. Redundancy Ranking: Tokens are ranked using a centrality score that accounts for both connectivity and connection quality. The most central (and thus most redundant) tokens are prioritized for removal.
  4. Training-Free Integration: Because it requires no parameter updates, SAINT can be integrated into existing architectures like CLIP, MAE, or LLaVA without fine-tuning.

Analysis of Token Dynamics

A core contribution of this work is the identification of a universal three-stage process in transformer layers, termed the Aligner-Explorer-Aggregator cycle. By analyzing metrics such as token similarity ($S$) and entropy ($H$) across model depth, the authors found that tokens are highly redundant in the early "aligner" stage, diversify during the middle "explorer" stage, and converge again in the final "aggregator" stage.

Evolution of token metrics across transformer layers, highlighting the Aligner-Explorer-Aggregator stages.

This insight suggests that aggressive pruning in the very first layers is most effective. Pruning during the "aligner" stage removes redundant information before it undergoes expensive processing in the middle layers, maximizing computational savings while preserving the essential features of the image.

Experimental Results

The researchers evaluated SAINT across standard benchmarks including ImageNet-1K for classification and a suite of multimodal tasks (MME, POPE, GQA, VizWiz).

ViT and VLM Performance

On the massive ViT-H/14 model, SAINT doubled throughput with only a 0.6% drop in ImageNet accuracy. In the multimodal domain, SAINT allowed LLaVA-13B to operate with 75% fewer visual tokens. This reduction enabled the 13B model to achieve latency comparable to the smaller LLaVA-7B while maintaining its superior reasoning capabilities.

Performance comparison of SAINT (Similarity-Based Drop/Merge) against baseline pruning strategies across various ViT and VLM backbones.

As shown in the comparative analysis, SAINT's similarity-based approaches consistently outperform attention-based pruning and random dropping. The "Similarity-Based Drop" and "Merge" variants maintain higher accuracy across varying pruning depths and thresholds, demonstrating that token similarity is a more robust metric for redundancy than attention weights in early layers.

Conclusion and Future Work

SAINT provides a highly efficient, training-free solution for scaling ViTs and VLMs. By capitalizing on the Aligner-Explorer-Aggregator dynamics, it offers a superior accuracy-vs-throughput trade-off compared to existing methods. Currently, SAINT is text-agnostic, meaning it prunes visual tokens without considering the user's specific query. Future research into "query-aware" pruning could further refine the selection process, ensuring that even small, task-relevant objects are preserved during extreme compression scenarios.

Summary updated

BibTeX
@misc{jeddi2025similarity,
  author = {Ahmadreza Jeddi and Negin Baghbanzadeh and Elham Dolatabadi and Babak Taati},
  title = {{Similarity-Aware Token Pruning: Your VLM but Faster}},
  year = {2025},
  eprint = {2503.11549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.11549},
}
Figure