Large Multimodal Models (LMMs) and Vision Transformers (ViTs) face a severe computational bottleneck: high-resolution images and videos generate hundreds or thousands of visual tokens. Because the computational cost of the self-attention mechanism scales quadratically with sequence length, processing all of these tokens makes inference slow and resource-intensive.
To alleviate this, early efficiency methods relied on token pruning—identifying and permanently deleting "unimportant" or redundant tokens (such as background patches) to speed up the network. However, hard token pruning introduces a critical flaw: destructive and irreversible information loss. Dropping tokens based on naive heuristics frequently discards sparse, fine-grained, or long-tail details that are essential for complex downstream reasoning.
To build faster models without sacrificing accuracy, modern research has shifted from destructive pruning to information-preserving token reduction. Understanding why naive pruning fails reveals the intuition behind these newer, more resilient architectures.
Why Naive Pruning Fails
To understand why simple token deletion degrades model performance, we must look at how transformers process context and how humans interact with multimodal models. Naive pruning strategies typically suffer from three core failure modes.
1. Saliency is Task-Dependent (Query Blindness)
Traditional pruning methods often evaluate token importance purely based on visual features or initial attention maps. However, an image patch that appears to be background noise to a general vision encoder might contain the exact information required to answer a specific user prompt.
As noted by Yi Chen et al. in Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information, a visual patch representing a tiny, non-salient object might be the sole target of a semantic query. When token reduction is prompt-agnostic, as Miao Liu et al. argue in Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction, the system risks prematurely discarding visual details that are critical for answering a particular question, leading to catastrophic failure modes on knowledge-intensive tasks.
2. The Redundancy Paradox
Intuitively, selecting tokens with the highest attention scores should preserve the most important information. In practice, this greedy selection strategy creates a paradox: it retains highly redundant representations while discarding diverse contextual cues.
Dmitrii Marin et al. demonstrate in Token Pooling in Vision Transformers that multiple high-scoring tokens often represent the exact same visual component (e.g., ten distinct tokens all representing parts of a central foreground object). Relying solely on importance scores wastes computational resources on duplicate representations while completely erasing lower-ranked, unique background features. This greedy dropping mechanism destroys the global semantic diversity of the scene, a phenomenon highlighted by Sifan Long et al. in Beyond Attentive Tokens: Incorporating Token Importance and Diversity for Efficient Vision Transformers.
3. The Irreversibility of Early-Stage Decisions
To maximize computational savings, models must reduce tokens as early in the network as possible. However, the initial layers of a deep network lack semantic depth.
As Feng Wu et al. explain in Conical Visual Concentration for Efficient Large Vision-Language Models, pruning visual tokens before the model has established a sufficient global context makes it impossible to accurately distinguish between true background noise and task-relevant features. Once a token is deleted, the information is permanently lost; it cannot be recovered or utilized by deeper layers of the network (Xuwei Xu et al. in GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation).
Mechanisms for Information Preservation
To circumvent the destructive nature of hard pruning, recent architectures employ alternative paradigms that compress the token space while safeguarding the underlying information.
Naive Pruning:
[Token A] [Token B] [Token C] ---> Drop B & C ---> [Token A] (B & C are lost)
Token Merging / Aggregation:
[Token A] [Token B] [Token C] ---> Cluster/Merge ---> [Token A] [Token B+C] (Context retained)
Token Squeezing / Propagation:
[Token A] [Token B] [Token C] ---> Squeeze B&C into A ---> [Token A + features of B&C]
Token Merging and Aggregation
Instead of dropping low-scoring or background tokens, models can group similar tokens together using clustering algorithms (such as K-Means or graph-based matching). This preserves the aggregate information of the entire visual field in a lower-resolution format rather than creating absolute information gaps.
For instance, Sam Pollard et al. show in Video, How Do Your Tokens Merge? that merging spatio-temporal patches instead of dropping them preserves subtle motion cues and background context that are vital for video understanding. Similarly, Lianyu Hu et al. leverage this concept in iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models by recycling information from redundant patches into a smaller set of representative tokens, maintaining high accuracy even at extreme compression ratios. To prevent the accidental blurring of unique features during this process, single-author work by Chau Tran in Accelerating Transformers with Spectrum-Preserving Token Merging introduces a graph-based energy score to explicitly safeguard unique, informative tokens before any clustering occurs.
Information Squeezing and Propagation
Another strategy is to harvest the features of tokens slated for removal and inject them into the remaining tokens. This ensures that the retained tokens act as "proxies" or carriers for the discarded context.
In Joint Token Pruning and Squeezing Towards More Aggressive Compression of Vision Transformers, Siyuan Wei et al. introduce a mechanism that "squeezes" the semantic context of discarded background patches into the remaining foreground tokens. By doing so, the network retains a comprehensive understanding of the environment without paying the computational cost of processing the background tokens individually. Similarly, the graph-based approach in GTP-ViT propagates the features of pruned tokens to their spatial neighbors, ensuring that no critical data vanishes from the network entirely.
Conditional Diversity and Query Guidance
To resolve query blindness, modern token reduction techniques incorporate the user's text prompt directly into the compression step, ensuring that diversity is maintained relative to the task context.
Qizhe Zhang et al. address this in Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs by framing token selection as a Determinantal Point Process (DPP). By balancing raw visual representation with explicit instruction relevance, their method ensures that the remaining tokens maximize conditional diversity. This retains the subtle, long-tail details required for specific queries while still compressing redundant background data.
When text guidance is paired with a representative subset selection strategy—such as the Max-Min diversity formulation proposed by Saeed Ranjbar Alvar et al. in DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models—the model avoids the accuracy drops typical of high-ratio pruning, maintaining a faithful, compressed representation of the entire scene.