Managing Visual Token Redundancy in Multimodal Models

862 concepts640 papers8 proposals

Multimodal Large Language Models (MLLMs) have achieved remarkable success by treating visual data as a sequence of tokens, similar to text. However, this approach introduces a significant "visual token tax." When a high-resolution image is decomposed into patches, it can generate hundreds or even thousands of tokens. As noted by Minbin Huang, because the language model is the most resource-intensive component of the system, this explosion of tokens leads to prohibitive computational overhead, high inference latency, and massive memory consumption.

The core challenge is that while high resolution is necessary for fine-grained tasks like document parsing or localization, much of the resulting visual data is redundant. Researchers are now developing strategies to identify and compress this redundancy without sacrificing the model's "vision."

The Bottleneck of Visual Token Explosion

In standard architectures like LLaVA, every visual patch from an encoder is passed directly to the LLM. Haogeng Liu et al. argue that this creates a massive computational burden, particularly as the number of tokens increases memory usage in the Key-Value (KV) cache. This expansion leads to GPU memory exhaustion and slows down the prefilling stage of inference.

The problem is exacerbated in video analysis and multi-image reasoning. For instance, Shizhou Huang et al. highlight that mapping specific image regions to textual entities in social media posts requires fine-grained grounding, which becomes computationally expensive when multiple high-resolution images are involved. Similarly, the authors of LongVILA-R1 note that reinforcement learning for video models is often hindered by the redundant re-encoding of high-dimensional visual tokens during policy optimization.

Pruning vs Merging

To mitigate this overhead, researchers generally follow two paths: removing tokens (pruning) or combining them (merging).

Adaptive Pruning

Pruning identifies "unimportant" tokens and discards them. Mark Endo et al. discovered that early pruning often biases models against specific image regions, such as the top of an image, which can hurt localization tasks. To solve this, they proposed FEATHER, a multi-stage pruning strategy. Other methods, like ATP-LLaVA by Xubing Ye et al., adaptively decide the pruning ratio based on the specific image and the model layer, allowing for deeper compression where redundancy is highest.

Redundancy-Aware Merging

Rather than simply deleting information, merging strategies aggregate similar tokens. Zhongwei Wan and Ziang Wu introduced LOOK-M, which merges redundant visual data in the KV cache to reduce the memory footprint by up to 95%. The authors of DART argue that "duplication matters more than importance," suggesting that pruning should focus on removing repetitive information rather than just low-attention tokens. By focusing on redundancy, they achieved a 3x speedup in the prefilling stage.

Learning Where to Look

A more sophisticated approach involves teaching the model to dynamically allocate its computational budget. Instead of processing every patch at high resolution, models can "zoom in" only when necessary.

  • Hierarchical Extraction: Anthony Fuller et al. proposed LookWhere?, which uses a lightweight low-resolution "selector" to identify important regions before a high-capacity "extractor" processes them at high resolution.
  • Reinforcement Learning: Senqiao Yang et al. introduced VisionThink, where a model starts with a low-resolution view and autonomously decides whether to request high-resolution patches to answer a specific question.
  • Semantic Routing: Long Cui et al. developed ViCO, which uses a lightweight router to identify complex semantic regions for high-resolution processing while compressing simpler background patches.

System-Level Optimization

Beyond algorithmic pruning, researchers are addressing the physical bottlenecks of token processing. Kunxi Li et al. proposed FlowMM, a framework that merges KV cache entries based on cross-modal information flow, speeding up text generation by up to 1.8x. To handle the stalls caused by heavy visual encoding, Zhicheng Li et al. introduced SpaceServe, which uses spatial multiplexing to run encoders and decoders simultaneously on partitioned GPU resources, significantly reducing token generation latency.

These diverse approaches—ranging from training-free pruning like FlowCut to progressive distillation methods like EPIC by Zichen Wen et al.—all point toward a shared goal: breaking the linear relationship between visual resolution and computational cost. By treating visual tokens as a sparse and redundant resource rather than a dense requirement, the field is moving toward MLLMs that can "see" high-resolution details without the associated "token tax."

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 522 papers
  2. Version 1 · 2026-10-04 · 869 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_visual_token_redundancy_visual_token_exp_968e385d,
  title = {visual token redundancy (visual token explosion)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-token-redundancy-visual-token-explosion/rdr_968e385d162e4f4183358daf6fd9b0e8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_visual_token_redundancy_visual_token_exp_968e385d_v2,
  title = {visual token redundancy (visual token explosion)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/visual-token-redundancy-visual-token-explosion/rdr_968e385d162e4f4183358daf6fd9b0e8/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake