Training-Free Visual Token Reduction

26 concepts26 papers2 proposals

Vision Transformers (ViTs) and Multimodal Large Language Models (MLLMs) process images and videos by breaking them down into a grid of local patches, each treated as an individual token. While this approach provides a uniform interface for visual data, it introduces a severe computational bottleneck: an image can easily generate hundreds of tokens, and a video can generate thousands. Because the computational complexity of the self-attention mechanism scales quadratically ($O(N^2)$) with the sequence length $N$, processing these visual tokens consumes massive amounts of memory and compute during inference.

To solve this problem, a powerful paradigm has emerged: training-free, plug-and-play visual token reduction. Instead of undergoing expensive architectural modifications or retraining cycles, researchers have found that we can dynamically prune or merge redundant visual tokens during the forward pass.

This approach works because pre-trained models already possess highly structured attention maps and embedding spaces. By inserting non-parametric, deterministic mathematical layers directly into existing models, we can exploit this latent structure to accelerate inference with virtually no loss in accuracy.


The Core Insight: Inherent Visual Redundancy

Why can we discard or combine visual tokens without breaking a model's performance? Visual data contains immense spatial and temporal redundancy. In a typical image, large regions (like a clear blue sky or a flat wall) contain uniform semantic information.

Furthermore, the deep layers of a transformer do not treat all tokens equally. As Chen et al. demonstrate in An Image is Worth 1/2 Tokens After Layer 2, large vision-language models spend enormous compute processing visual tokens that their deeper layers almost entirely ignore. By analyzing existing attention patterns, the authors show that a model naturally concentrates its attention on a sparse subset of informative tokens after the first few layers.

This realization allows for immediate inference-time optimization. For example, Arif et al. introduce a framework called HiRED in HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models, which uses the vision transformer's own internal attention weights to filter out redundant tokens before they ever reach the heavy large language model (LLM) decoder.


Strategy 1: Strategic Token Pruning

The simplest way to reduce token count is to drop uninformative tokens entirely. However, deciding which tokens to drop requires careful balance. If a model only chases the most highly highlighted tokens, it risks losing the broader context of the scene.

Balancing Saliency and Coverage

To prevent the loss of holistic context, Deng et al. propose SCOPE in SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs. Instead of relying solely on saliency (how strongly a token is attended to), the authors balance it with a coverage metric to ensure semantic diversity is preserved across the entire image. This philosophy is shared by Zou et al. in Don't Just Chase “Highlighted Tokens” in MLLMs: Revisiting Visual Holistic Context Retention, who argue that retaining background context is vital for visual reasoning tasks.

Geometric and Graph-Based Diversity

Other pruning methods look at the geometric distribution of tokens in the embedding space. In DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models, Alvar et al. formulate token selection as a Max-Min Diversity Problem. By computing cosine similarities directly on the pre-trained embedding space, they select a subset of tokens that maximizes geometric diversity, allowing the model to retain high accuracy even when discarding up to 80% of the visual tokens.

[Standard Grid] --------> [Similarity Graph] --------> [Pruned Subset]
(High Redundancy)        (Connects close tokens)       (Retains core geometry)

Taking a structural view, Wu et al. develop G-Prune in What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph, which models token relationships as a graph to identify and retain the most representative nodes across both foreground and background elements.

Incorporating Low-Level Features

While most methods rely on high-level semantic embeddings, Ouyang et al. look to classical computer vision in Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruning. Their framework, LVTP, combines high-level transformer attention maps with low-level edge detection. This ensures that tokens containing critical structural boundaries and fine details are never accidentally pruned, which is particularly beneficial for dense prediction tasks like semantic segmentation.


Strategy 2: Information-Preserving Merging

Pruning completely discards tokens, which risks throwing away subtle but necessary details. An alternative approach is token merging, which clusters similar tokens together and aggregates their features.

[Token A] \
           --> [Compute Similarity] --> [Weighted Average] --> [Merged Token AB]
[Token B] /

Spatial-Aware Clustering

Early work by Liang et al. in Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning proved that inserting non-parametric clustering and reconstruction layers into pre-trained models could achieve significant computational savings without any weight optimization. Building on this, Li et al. introduce AiluRus in AiluRus: A Scalable ViT Framework for Dense Prediction, utilizing a spatial-aware clustering algorithm that merges redundant tokens in flat, uniform areas while preserving distinct tokens at object boundaries.

To prevent spatial distortion during the merging process, Huang et al. develop ToSA in ToSA: Token Merging with Spatial Awareness. ToSA explicitly incorporates depth information alongside visual feature similarity, ensuring that tokens are only merged if they are spatially and structurally coherent in the 3D scene.

Structural Collapsing and Recycling

Instead of clustering arbitrary groups of tokens, some methods enforce rigid structural reductions to maintain hardware efficiency. Su et al. introduce Token Adapter in Removing Rows and Columns of Tokens in Vision Transformer Enables Faster Dense Prediction Without Retraining, which strategically merges entire rows and columns of redundant tokens and reconstructs them later in the network, preserving the standard tensor shapes required for fast hardware execution.

Rather than choosing between pruning and merging, Hu et al. combine the best of both worlds in iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models. Their approach recycles information from redundant patches by merging their features into the remaining representative tokens instead of dropping them entirely, cutting input tokens by over 66% while maintaining high accuracy.


Strategy 3: Spatiotemporal Budgets for Video

When moving from static images to video, token explosion worsens by an order of magnitude due to the addition of the temporal dimension. Consecutive video frames are highly correlated, meaning the redundancy is both spatial (within a frame) and temporal (across frames).

Frame t   [ Patch A ] [ Patch B ]
               \           /
                \         /   (Temporal & Spatial Merging)
                 v       v
Frame t+1 [ Patch A'] [ Patch C ]

To tackle this, Zhuang et al. propose ST3 in ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming. ST3 dynamically trims tokens across both model layers (spatial) and the generation process (temporal), slashing computational costs and Key-Value (KV) cache memory usage by over 50% during the inference prefill stage.

Adaptive Allocation

Uniform compression across a video often fails because some scenes contain rapid motion while others are static. Wang et al. address this in AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding by adaptively allocating token compression budgets across both time and model layers based on the video's dynamic content.

Similarly, Luo et al. introduce ST-GridPool in Enhancing Visual Token Representations for Video Large Language Models via Training-free Spatial-Temporal Pooling and Gridding, which pairs hierarchical temporal gridding with token-norm spatial prioritization to compress video sequences deterministically.

Finally, to handle long-context video understanding without rigid frame-by-frame constraints, Fan et al. develop FlashVID in FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging. FlashVID utilizes a dynamic, tree-based merging strategy that allows the model to aggregate spatiotemporal tokens fluidly across arbitrary frame boundaries, enabling models to process up to 10× more video frames within the same computational budget.


Summary

Training-free visual token reduction turns an architectural flaw—the quadratic cost of processing highly redundant visual grids—into an optimization opportunity. By leveraging the geometric and attention properties already present inside pre-trained transformers, these plug-and-play methods allow developers to trade a fraction of a percent of accuracy for massive gains in throughput and memory efficiency, all without spending a single dollar on retraining.

Key Foundational Work

Expediting Large-Scale Vision Transformer for Dense Prediction without Fine-tuning (2022) — contributed 1 concept elements to this direction.
AiluRus: A Scalable ViT Framework for Dense Prediction (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_plug_and_play_token_reduction_9dbfa9ae,
  title = {plug-and-play token reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/plug-and-play-token-reduction/rdr_9dbfa9ae97a540c1ad66eaac6aa73a3d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_plug_and_play_token_reduction_9dbfa9ae_v1,
  title = {plug-and-play token reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/plug-and-play-token-reduction/rdr_9dbfa9ae97a540c1ad66eaac6aa73a3d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake