Large Vision-Language Models (LVLMs) like LLaVA are computationally expensive because they treat images as long sequences of visual tokens. Since Transformer complexity grows quadratically with sequence length, processing hundreds of image patches alongside text creates a massive bottleneck.
However, recent research reveals that most of these visual tokens are redundant. As information flows through the model, the "visual" work is completed early on, and the deeper layers shift toward textual reasoning. By identifying where and how this information concentrates, researchers have developed methods to prune or merge up to 90% of visual tokens without losing accuracy.
The Non-Uniformity of Visual Information
The core intuition behind efficient LVLMs is that visual tokens do not contribute equally across all layers of a model. In the initial layers, the model needs a high density of tokens to establish global context and extract fine-grained features. But as the data moves deeper, the model’s attention narrows.
Feng Wu and Long Xing describe this as a "pyramid" structure in PyramidDrop. They found that shallow layers are highly sensitive to token loss, while deeper layers are robust. Similarly, Qiong Wu and Weihao Ye demonstrate in FitPrune that deeper layers exhibit a much higher tolerance for token removal, allowing for aggressive pruning in the final stages of the network.
This transition is often abrupt. Yihao Quan and Liang Xie identified a "cliff layer" where visual information flow diverges. Beyond this point, image tokens stop contributing meaningfully to the reasoning process, and the model relies almost entirely on internal textual logic to generate an answer.
Mapping the Information Flow
To prune tokens safely, we must distinguish between "background" noise and "pivotal" tokens. Traditional attention scores can be unstable, so researchers have turned to modeling the "flow" of information across the entire architecture.
In FlowCut, Jintao Tong and Wenwei Jin argue that redundancy is a dynamic state. As a model focuses on "hub" tokens in deeper layers, peripheral tokens become redundant. Yuzhang Shang and Bingxin Xu found in freePruner that a single token can facilitate up to 70% of the total information flow in middle layers.
This concentration of information means that most visual tokens act as "donors." Once they have passed their semantic content to a few dominant tokens or text tokens, their presence in the sequence provides diminishing returns. Shaolei Zhang takes this to the extreme with LLaVA-Mini, showing that by "pre-fusing" visual data, a model can function effectively with just one vision token per image.
Mechanisms for Token Reduction
Researchers use several strategies to exploit this redundancy without requiring expensive retraining:
- Adaptive Pruning: Xubing Ye and Yansong Tang developed ATP-LLaVA, which adaptively determines pruning ratios for each specific image. This ensures that complex images retain more detail while simple ones are compressed further.
- Token Merging: Instead of deleting tokens, VisionZip by Senqiao Yang and Yukang Chen selects dominant tokens and merges similar ones together, preserving more information than simple pruning.
- Spatial Diversity: Simple attention-based pruning often suffers from "clumping," where the model keeps many identical tokens from the same object. Bozhi Luan and Wengang Zhou introduced AdaptPrune, which uses Non-Maximum Suppression (NMS) to ensure the pruned set of tokens remains spatially diverse and unique.
Efficiency Without Sacrifice
The result of these methods is a significant reduction in the "computational tax" of multimodal AI. By mimicking the human "glance-then-gaze" process—as proposed by Juan Chen in Glance2Gaze—models can achieve up to a 2x speedup in inference and a 90% reduction in tokens.
Because these techniques are often training-free and "plug-and-play," they allow existing large-scale models to run on humbler hardware. The evidence across these 24 papers suggests that for high-level reasoning, the vast majority of visual tokens are not just redundant—they are unnecessary.