Content Adaptive Token Reduction

414 concepts297 papers1 proposal

The Inefficiency of Fixed-Grid Tokenization

Modern Vision Transformers (ViTs) and Multimodal Large Language Models (MLLMs) process visual inputs by breaking them down into a sequence of discrete tokens. In standard architectures, this tokenization follows a rigid, "one-size-fits-all" paradigm: an image is sliced into a uniform grid of square patches (such as $14 \times 14$), and a video is sampled at a fixed frame rate.

While computationally convenient, this uniform approach ignores the underlying semantic structure of visual data. A clear blue sky, a blank wall, or a static video background contains far less information than a dense paragraph of text, a human face, or a high-motion action sequence. Yet, standard architectures allocate the exact same number of tokens—and consequently, the same amount of computational FLOPs—to both simple and complex regions. Because the computational complexity of self-attention scales quadratically with sequence length, this uniform allocation imposes a massive efficiency tax.

Content-adaptive token reduction shifts the computational bottleneck from a fixed spatial grid to a variable budget determined by the information density of the input. By dynamically pruning or merging tokens based on the complexity of the content, models can process simple inputs at extreme compression rates while preserving fine-grained representations for detail-rich scenes.

Gauging Complexity: How Models Measure Information Density

To reduce tokens dynamically without sacrificing accuracy, a model must first determine which tokens are redundant and which are critical. Researchers have developed several distinct mechanisms to evaluate this on the fly.

Attention-Guided Metrics

One of the most direct signals of token importance is the model's own internal attention matrix. In a transformer layer, the attention weights reflect how much information a token exchanges with others. If a token receives low attention across the board, it contributes little to the final representation.

The authors of An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models demonstrate that early layers in an MLLM can reliably identify uninformative visual tokens that deeper layers almost entirely ignore. By implementing a plug-and-play pruning mechanism after these initial layers, they discard redundant visual data before it reaches the heavy autoregressive phase. Similarly, Kazi Hasan Ibn Arif et al. leverage attention scores in HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models to dynamically allocate token budgets across different image regions, ensuring that high-content areas retain their density while uniform backgrounds are aggressively thinned.

Information Theory and Statistical Criteria

Rather than relying purely on attention heuristics, some frameworks treat token reduction as a formal model selection problem rooted in information theory. For instance, Marius Aasan et al. introduce a method in Differentiable Hierarchical Visual Tokenization that applies information criteria to prune hierarchical image partitions. By balancing representation fidelity against computational complexity, their approach automatically determines the optimal number of tokens for a given scene without requiring manual hyperparameter tuning.

Temporal and Semantic Dynamics

In video processing, complexity is driven by motion and temporal transitions. Sampling frames uniformly results in severe over-tokenization during static scenes. To combat this, the authors of Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understanding introduce a strategy that adjusts the compression rate of visual data based on scene change scores. High-activity segments are processed with fine-grained intervals to capture rapid motion, while static or redundant segments are compressed into larger intervals to optimize throughput.

Mechanics of Reduction: Pruning vs. Merging

Once a model has quantified information density, it must execute the reduction. The literature splits this operation into two primary paradigms: token pruning and token merging.

Token Pruning:
[Token A (High Info)]  ---> Keep ---> [Token A]
[Token B (Low Info)]   ---> Drop
[Token C (High Info)]  ---> Keep ---> [Token C]

Token Merging:
[Token A (Background)] \
[Token B (Background)]  ---> Cluster & Average ---> [Token AB (Merged Background)]
[Token C (Foreground)]  -------------------------> [Token C  (Preserved Foreground)]

Token Pruning

Pruning completely removes low-importance tokens from the sequence. Xubing Ye demonstrates in ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models that adaptively dropping visual tokens based on image complexity and model depth can eliminate up to 75% of visual tokens with negligible performance loss. This instance-wise adaptation ensures that simple scenes are aggressively thinned while complex layouts remain intact.

In the domain of state space models, Tien-Yu Chi introduces QuarterMap: Efficient Post-Training Token Pruning for Visual State Space Models, which strategically removes redundant spatial activations before the scanning process and restores them via simple upsampling later, bypassing the need for expensive model retraining.

Token Merging

Completely discarding tokens risks losing subtle background context that might be relevant to downstream reasoning. Token merging addresses this by grouping semantically similar tokens into a single representative vector.

Youwei Liang explores this in EViT: Expediting Vision Transformers via Token Reorganizations, where the most attentive tokens are kept as-is, while the remaining redundant tokens are fused into a single token that aggregates their features. To make this process content-adaptive, Zhenhailong Wang et al. develop DyMU: Dynamic Merging and Virtual Unmerging for Efficient Variable-Length VLMs. Their framework uses similarity thresholds to dynamically compress redundant regions while preserving high-resolution details in information-dense areas like text or faces, utilizing a mathematical "virtual unmerging" trick to maintain compatibility with standard model layers.

Dynamic Tokenization at the Source

While post-hoc pruning and merging operate on tokens after they have been generated by a standard vision encoder, a parallel line of research embeds content-adaptivity directly into the tokenization process itself. These models learn to encode visual data into a variable number of tokens from the very beginning.

The authors of CAT: Content-Adaptive Image Tokenization introduce a nested VAE architecture that dynamically adjusts its compression ratio based on image complexity. This allows the framework to redistribute tokens across a dataset, allocating more tokens to detail-rich images and fewer to simple ones, optimizing the overall reconstruction quality under a fixed computational budget.

For autoregressive video modeling, where the quadratic cost of long sequences is particularly severe, Minghao Yang et al. propose VaporTok: RL-Driven Adaptive Video Tokenizer with Prior & Task Awareness. By combining a probabilistic taildrop mechanism with reinforcement learning, the tokenizer dynamically cuts down token counts by up to 70% for visually simple sequences.

Similarly, Wilson Yan et al. develop ElasticTok: Adaptive Tokenization for Image and Video, which uses a random masking strategy during training to enable a single model to encode images and videos into a variable number of tokens, matching the token budget directly to scene complexity.

Why It Works: Instance-Wise and Layer-Wise Adaptation

The success of content-adaptive token reduction relies on exploiting two distinct dimensions of redundancy: instance-wise and layer-wise sparsity.

  • Instance-wise adaptation acknowledges that different inputs require different amounts of representational capacity. A uniform pruning strategy treats a complex document and a simple landscape identically, leading to catastrophic information loss in the former or wasted compute in the latter. By making the token budget a function of the input's semantic complexity, models maintain high fidelity where it matters most.
  • Layer-wise adaptation exploits the hierarchical nature of deep networks. Early layers in a transformer extract low-level features (such as edges and textures) that require high spatial precision. As the data flows deeper, the network abstracts these features into high-level concepts. Consequently, the required token density naturally decreases as depth increases.

By combining instance-adaptive thresholds with layer-progressive reduction, these frameworks successfully break the rigid constraints of uniform grids, allowing modern transformers to scale efficiently to long-context, high-resolution multimodal tasks.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 262 papers
  2. Version 1 · 2026-10-04 · 453 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_token_reduction_b99d6b2b,
  title = {dynamic token reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-token-reduction/rdr_b99d6b2bf2d748eca48bf5a79be249b5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_dynamic_token_reduction_b99d6b2b_v2,
  title = {dynamic token reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-token-reduction/rdr_b99d6b2bf2d748eca48bf5a79be249b5/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake