When a Large Vision-Language Model (LVLM) processes an image, it typically divides the visual input into a dense grid of patches, converting each patch into a visual token. While a sentence might require only a few dozen tokens, a single high-resolution image or video clip can easily generate hundreds or thousands of visual tokens.
Because the computational complexity of the self-attention mechanism scales quadratically with sequence length, these massive visual sequences overwhelm the language backbone. However, a growing body of research demonstrates that this computational burden is largely unnecessary: visual tokens possess extreme informational redundancy, allowing models to discard or merge up to 95% of them during inference with virtually no loss in performance.
The Modality Asymmetry Problem
To understand why visual token pruning works, we must first look at the fundamental difference between text and vision. Text is highly compressed and discrete; every word or subword token carries dense semantic weight. In contrast, visual data is spatially and temporally continuous. Neighboring patches in an image often represent the exact same background pixel values, carrying identical semantic information.
As a result, vision tokens exhibit a high degree of fault tolerance. For example, Andrey Gizdov et al. found that models can maintain roughly 80% of their task performance even when visual input is reduced to a tiny fraction of its original pixels by mimicking human foveated vision. Similarly, Rinu Boney demonstrated that models can maintain accurate perception even when the vast majority of image tokens are completely hidden. This high tolerance for degradation suggests that allocating equal precision and compute to every visual token is highly inefficient, a concept leveraged by the authors of MBQ to aggressively quantize visual tokens while preserving the precision of more sensitive language tokens.
The Lifecycle of a Visual Token
The redundancy of visual information is not static; it evolves as a sequence travels deeper through the transformer layers.
[Input Image] ──> [Early Layers: Alignment] ──> [Middle Layers: Pruning Window] ──> [Deep Layers: Pure Text Generation]
(Visual tokens map (Redundant background tokens (Visual info fully absorbed;
to text anchors) are discarded or merged) vision tokens become dead weight)
In the early layers of an LVLM, visual tokens are critical for cross-modal alignment, mapping raw sensory data to linguistic concepts. However, as the information flows deeper, its role changes. Yixian Shen et al. observed that visual tokens frequently become redundant in deeper layers because their semantic information is effectively "absorbed" into the textual stream or system prompts.
Once the text tokens have extracted the necessary context from the image, the model shifts its focus toward generating the linguistic response. At this stage, retaining the high-dimensional visual representations creates massive, unnecessary computational overhead. Liang Chen et al. leveraged this insight to create FastV, a plug-and-play method that discards redundant visual tokens after the first few layers, slashing FLOPs by up to 45%.
This depth-wise redundancy means that visual processing can be localized or even truncated. Jing Bi et al. identified specialized "visual attention heads" concentrated primarily in the early and middle layers, proving that late-stage attention heads can be bypassed. Taking this further, Hao Wu et al. introduced HiDrop, which defines a strict "vision processing window"—skipping computation in shallow layers, pruning aggressively in the middle, and exiting visual token processing entirely in deep layers.
Mechanisms of Reduction: Pruning vs. Merging
To exploit this redundancy without sacrificing accuracy, researchers use two primary strategies: pruning (deleting uninformative tokens) and merging (combining similar tokens).
Token Pruning
Pruning relies on identifying which tokens are non-essential. A common approach is to use the model's own attention weights as an indicator of importance. Kazi Hasan Ibn Arif developed HiRED, which uses the Vision Transformer's internal attention scores to drop redundant tokens before they ever reach the language model.
However, single-layer attention scores can be unstable. To address this, Jintao Tong et al. introduced FlowCut, which models the "information flow" dynamically across multiple layers. As the model focuses on central hub tokens in deeper layers, peripheral tokens are safely cut, achieving up to a 94.4% token reduction.
Token Merging
Pure pruning risks discarding fine-grained details if an important feature is misidentified. Merging mitigates this by clustering tokens and aggregating their representations. Chau Tran et al. proposed PITOME, which uses a graph-based energy score to identify clusters of similar data, merging redundant tokens while safeguarding unique, informative ones.
Similarly, for long-context scenarios, Kunxi Li et al. introduced FlowMM, which merges redundant visual data within the Key-Value (KV) cache rather than deleting it outright, preserving memory while accelerating text generation.
Context and Query-Aware Selection
How do we determine what makes a token "important"? Interestingly, a benchmark study by Tianfan Peng et al. revealed that simple random token pruning serves as a surprisingly effective baseline, often matching complex, specifically designed pruning strategies. This underscores just how pervasive visual redundancy is.
However, to achieve extreme compression ratios (above 90%) without losing accuracy, token reduction must become query-aware. A token representing a small patch of text in an image might be redundant for a general question about the scene, but it is vital if the user asks, "What does the sign say?"
[User Query: "What color is the car?"]
│
▼
[Image Tokens] ──> [Query-Vision Interaction (SVD/DPP)] ──> [Retained: Car Tokens (High Relevance)]
──> [Pruned: Sky/Tree Tokens (Low Relevance)]
To solve this, Qizhe Zhang et al. developed CDPruner, which balances general image representation with specific instruction relevance by maximizing "conditional diversity" via a Determinantal Point Process. By explicitly conditioning the token selection on the text prompt, it removes tokens that do not contribute to the specific task context.
In a similar vein, Harshithanjani Athi et al. introduced QViD, which applies Singular Value Decomposition (SVD) directly to the interaction matrix between text queries and image features, ensuring that only the most semantically relevant image regions are retained under aggressive compression.
The Noise Reduction Bonus
The most surprising insight from high-ratio token pruning is that reducing the number of visual tokens does not merely maintain performance—it frequently improves it.
When an LVLM is forced to attend to hundreds of uninformative background tokens, its attention weights become diluted. Highly repetitive or uninformative background tokens can act as "high-norm outliers" or sinks, distracting the language model and causing it to hallucinate.
The authors of DART (Duplication-Aware Reduction of Tokens) demonstrated that eliminating duplicate tokens in multimodal inputs directly reduces model hallucinations and improves reasoning accuracy. By removing the "noise" of excessive visual evidence, the language model can focus entirely on the core semantic facts. As shown by Yejie Guo et al., reducing the volume of input data to a minimal sufficient set prevents attention dilution, allowing spatial reasoning accuracy to increase precisely because the model has less irrelevant information to sift through.