Vision Transformers (ViTs) and Large Multimodal Models (LMMs) are computationally expensive because they treat every part of an image—from a central object to a blurry corner—as a distinct "token" of equal cost. To speed these models up, researchers initially tried pruning: simply deleting tokens deemed "unimportant."
However, pruning is destructive. As noted by Hao Zheng in CageViT, discarding background patches often destroys the global context or fine-grained details needed for complex reasoning. Even in linguistics, as Rohan Chavan explores in Curating Stopwords in Marathi, removing high-frequency "thin" tokens can degrade the structural information that Transformer attention mechanisms rely on.
The emerging consensus across recent research is a shift from pruning (deletion) to merging and recycling (compression).
The Problem with Deletion
When a model prunes a token, that information is gone forever. This creates a "fragility" in the model's understanding. For example, Yiqun Sun points out in ZipRerank that in dense documents like tables, a single omitted token might contain the only evidence needed to answer a query.
Traditional pruning assumes that low-attention scores equal zero utility. But as Lianyu Hu argues in iLLaVA, even less-salient regions contribute to the overall semantic "vibe" or background of an image. If you delete 70% of the tokens, you aren't just losing noise; you're losing the connective tissue of the scene.
Merging and Recycling Information
Instead of throwing tokens away, researchers are now "packing" the information from many redundant tokens into a few representative ones.
- Anchor and Summary Tokens: In LLaVA-PruMerge, Yuzhang Shang identifies "anchor" tokens and merges redundant neighbors into them. Similarly, CageViT keeps high-resolution "major" tokens but compresses "minor" background tokens into a few summary representations.
- Token Recycling: Lianyu Hu describes this in iLLaVA as a way to maintain accuracy while cutting input tokens by 66%. The goal is to ensure that the "essence" of the discarded data is still accessible to the model's later layers.
- Proxy Tokens: Penghao Wu introduces ProxyV, which uses a small set of "proxy" tokens to handle the heavy lifting of computation. This allows the model to reduce FLOPs significantly without losing the fine-grained detail required for tasks like OCR.
Text-Guided Importance
A key breakthrough in making this compression "lossless" is using the user's text prompt to guide the process. In a multimodal setting, "importance" is relative. If a user asks about a "red car" in the background, a general importance metric might have pruned those car tokens.
Chaoya Jiang (in TRIPS) and Hanxun Yu (in VisionTrim) use text-relevant selection to "rescue" tokens that are semantically related to the query. Pengfei Jiang takes this further in VISA by treating tokens as nodes in a graph, using text-guided attention to decide which nodes to aggregate.
Distillation and Long Context
This philosophy extends to the Key-Value (KV) cache in long-context models. Zhongwei Wan shows in LOOK-M that merging redundant visual data in the cache is far more effective than deleting it.
For extreme compression, Guanghui Qin introduces KV-Distill, which treats the process as a student-teacher task. The model learns to "pack" the essential information from a massive context into a tiny fraction of the space. This move from "selection" to "distillation" represents the ultimate version of context preservation: the information isn't just kept; it's intelligently reformulated for efficiency.