Transformer architectures face a fundamental scaling bottleneck: the computational complexity of self-attention scales quadratically ($O(N^2)$) with the number of tokens $N$. Whether processing high-resolution images, long video sequences, or extensive text documents, this "token explosion" rapidly exhausts memory and slows down inference.
To mitigate this, early efficiency methods relied on token pruning—identifying tokens with low attention scores and deleting them entirely from the sequence. However, hard deletion introduces a severe flaw: tokens deemed "unimportant" by an intermediate layer often contain subtle background context or complementary information that is vital for downstream reasoning. Completely discarding them causes models to suffer from performance drops, training instability, and hallucinations.
A powerful alternative paradigm has emerged: Context-Preserving Token Aggregation and Merging. Instead of treating token reduction as a binary choice of keeping or deleting, these methods treat it as a problem of lossy compression. By mathematically condensing redundant or low-importance tokens rather than throwing them away, they achieve massive speedups while preserving the global context of the sequence.
The Global Summary Token
The simplest way to transition from token deletion to token compression is to create an informational safety net. Instead of erasing low-attention tokens, we can fuse them into a single representative vector that remains in the sequence.
[Active Token 1] ──────> [Active Token 1]
[Active Token 2] ──────> [Active Token 2]
[Background 1] ──┐
[Background 2] ──┼───> [Global Summary Token]
[Background 3] ──┘
This approach was pioneered by Liang et al. in their work on EViT. In a Vision Transformer (ViT), they identify the top-$K$ most attentive visual tokens to keep as fine-grained features. Crucially, instead of discarding the remaining tokens, they aggregate them into a single summary token using a weighted average based on their attention scores. This single token acts as a low-resolution representation of the background, stabilizing training and maintaining global context.
This mechanism has been widely adapted across modalities:
- Multimodal LLMs: Song et al. developed TRIM, which uses CLIP scores to identify question-relevant image patches. It averages all non-selected visual tokens into a single aggregated summary token, allowing the language model to retain background context while slashing visual token counts by nearly 80%.
- Vision-Language Pre-training: Jiang et al. introduced TRIPS, which uses text-guided attention to select patches and aggregates the remaining ones via weighted fusion to improve model robustness.
- Signal Decomposition: The authors of Frequency-Aware Token Reduction for Efficient Vision Transformer frame this mathematically by separating unique, detailed "high-frequency" tokens from "low-frequency" tokens, which are compressed into a single "DC token" to mitigate the common problem of rank collapse in deep networks.
Clustering as Signal Downsampling
While a single summary token works well for preserving a uniform background, complex scenes require a more granular compression strategy. To achieve this, researchers treat the token sequence as discrete samples of a continuous signal and use clustering algorithms to group similar representations together.
Marin et al. introduced this perspective with Token Pooling. Instead of selecting tokens based on an importance score, they use clustering algorithms like K-Medoids to partition all tokens into a smaller set of representative centroids.
This highlights why clustering is fundamentally superior to top-$K$ importance pruning:
| Top-$K$ Importance Pruning | Clustering-Based Aggregation |
|---|---|
| Selects isolated tokens with the highest individual scores. | Groups tokens based on feature similarity. |
| Risk: May retain multiple highly redundant tokens from the same prominent object while completely erasing unique, lower-scoring background features. | Benefit: Merges redundant high-value features into a single centroid, freeing up budget to preserve a diverse summary of the entire input signal. |
To scale this without heavy computational overhead, Haurum et al. proposed Agglomerative Token Clustering (ATC), a parameter-free, bottom-up hierarchical clustering method that maintains spatial and structural consistency without requiring any fine-tuning. Similarly, the authors of Recoverable Compression utilize $K$-Nearest Neighbors to group secondary background tokens into a compressed set of centroids, acting as an informational safety net for multimodal language models.
Token Squeezing and Neighbor Propagation
An alternative to creating new summary tokens or cluster centroids is to "recycle" the information from discarded tokens by injecting their features directly into the tokens that are being retained.
[Token A (Keep)] ───> [Token A + f(Token B)]
[Token B (Drop)] ───┘
Wei et al. implemented this concept through Token Pruning and Squeezing (TPS). When a token is slated for deletion, its features are "squeezed" into a designated set of reserved host tokens via feature fusion. Because tokens identified as unimportant often contain complementary contextual data, fusing them prevents the steep performance drops typically caused by aggressive model compression.
Rather than performing a one-to-one merge, Xu et al. proposed GTP-ViT, which models the token sequence as a graph. When a node (token) is pruned, its features are propagated multilaterally across all its neighboring nodes, ensuring that the remaining tokens encapsulate a more complete representation of the original input space.
In the context of long-context multimodal inference, Wan developed MEDA, a framework that uses many-to-one token merging to mathematically integrate less significant visual tokens into their most relevant neighbors, maintaining global coherence across long sequences.
Compressing the KV Cache in Long-Context Inference
The benefits of context-preserving token reduction extend beyond the forward pass of vision encoders; they are increasingly critical for managing the Key-Value (KV) cache during Large Language Model (LLM) inference. As an LLM generates text, it stores the Key and Value vectors for all past tokens to avoid recomputation. For long documents or videos, this cache quickly grows large enough to overwhelm GPU memory.
Standard cache eviction policies simply drop old or low-attention KV pairs. However, just like pruning in vision models, hard eviction causes the model to forget past context.
Nawrot et al. addressed this with Dynamic Memory Compression (DMC). DMC retrofits LLMs to compress the KV cache on the fly by learning to merge redundant tokens through weighted averaging. Instead of permanently deleting history, it maintains a running summary of the sequence history.
Standard Eviction: [Token 1][Token 2][Token 3] ───> [Token 1] (Dropped 2 & 3)
DMC Compression: [Token 1][Token 2][Token 3] ───> [Token 1][Token 2+3] (Compressed 2 & 3)
In the multimodal domain, Wan et al. introduced LOOK-M, an optimization strategy that treats KV cache eviction as a form of lossy compression. By prioritizing text tokens and merging redundant visual tokens rather than deleting them, LOOK-M mitigates model hallucinations in long-context tasks.
To further bridge the gap between compressed and full-cache behavior, the authors of RestoreKV insert a small number of learned "restore tokens" into the sequence. These tokens are trained to aggregate information from the entire context immediately before an aggressive eviction occurs, acting as a highly distilled proxy for the deleted history.
Summary of the Intuition
The shift from token pruning to token aggregation relies on a simple realization: information density is distributed, not binary. A token with a low attention score is rarely completely useless; it contains background context that anchors the high-attention tokens. By shifting from hard truncation to lossy compression—whether through global summary tokens, similarity clustering, or neighbor propagation—modern architectures can shed up to 80% of their computational load without losing their memory of the global context.