Transformers have a well-known computational bottleneck: the self-attention mechanism scales quadratically $O(N^2)$ with the number of tokens $N$. To make these models faster and more memory-efficient, a natural strategy is token reduction—merging, clustering, or compacting redundant tokens (such as uniform background patches in an image or repetitive words in a text document).
However, simply dropping or naively averaging tokens introduces a subtle mathematical flaw that degrades model performance. When multiple tokens are compressed into a single representative token, the standard attention mechanism treats that new token as having a weight of exactly one. This creates an "attention dilution" problem, where large compressed regions lose their collective voting power.
To solve this, a shared research direction has emerged: Proportional Attention Weighting (or attention mass preservation). By tracking the "mass" or cardinality of compressed tokens and injecting it directly into the attention calculation, models can achieve massive speedups while remaining mathematically aligned with their uncompressed counterparts.
The Dilution Problem in Softmax Attention
To understand why naive token reduction fails, consider how standard attention computes the relationship between a query $q_i$ and a set of keys $k_j$. The attention weights are determined by a softmax operation over the dot products:
$$A_{ij} = \frac{\exp(q_i \cdot k_j / \sqrt{d})}{\sum_{m} \exp(q_i \cdot k_m / \sqrt{d})}$$
Imagine a Vision Transformer processing an image of a bird against a clear blue sky. The sky might consist of 50 highly redundant tokens. If an algorithm merges those 50 background tokens into a single "sky" token, the total number of tokens drops, accelerating the model.
However, during the next attention layer, that single sky token now contributes only one exponential term to the denominator ($\sum_m \exp(\cdot)$). Before compression, the 50 sky tokens collectively dominated the denominator, anchoring the global context. After compression, the sky's influence is artificially slashed by a factor of 50. The model suddenly over-indexes on unmerged foreground details, distorting its internal representations.
The Mathematical Intuition: Preserving Attention Mass
The core insight of proportional attention weighting is that we can perfectly restore this lost balance by leveraging the algebraic properties of the exponential function.
Suppose a cluster contains $s$ identical or highly similar tokens. If left unmerged, their collective contribution to the softmax denominator would be:
$$\sum_{m \in \text{cluster}} \exp(q_i \cdot k_m / \sqrt{d}) = s \cdot \exp(q_i \cdot \tilde{k} / \sqrt{d})$$
where $\tilde{k}$ is the representative key of the cluster. If we compress the cluster into a single token, we can preserve its original "attention mass" by multiplying its exponential score by $s$. Because $s = \exp(\log s)$, we can rewrite this multiplier as a simple additive bias inside the exponent:
$$s \cdot \exp(q_i \cdot \tilde{k} / \sqrt{d}) = \exp\left(q_i \cdot \tilde{k} / \sqrt{d} + \log s\right)$$
By adding a logarithmic compensation term ($\log s$) or a proportional bias directly to the attention logits, a single compressed token exerts an influence exactly proportional to the volume of data it represents.
Naive Merging:
[Token] (s=10) ───> Logit: (q·k) ───> Softmax Weight: exp(q·k) (Mass Lost!)
Proportional Weighting:
[Token] (s=10) ───> Logit: (q·k) + log(10) ───> Softmax Weight: 10 · exp(q·k) (Mass Preserved)
This elegant trick allows token reduction to happen dynamically without requiring any retraining. Daniel Bolya et al. pioneered this approach in Token Merging: Your ViT But Faster, demonstrating that introducing a size-based weight into the softmax calculation allows Vision Transformers to double their throughput while maintaining high accuracy.
This logarithmic correction has since been formalized across various domains. For instance, Hanruo Zhu et al. utilized it in BccT: an efficient transformer model for blood cell classification to ensure that fused image patches maintain their statistical significance and global context during aggressive token reduction.
Preserving the Value Stream
Attention is a two-part routing mechanism: softmax weights determine where information flows, but the Value (V) vectors dictate what information is actually aggregated. If keys are compressed, the corresponding values must be handled with equal mathematical care.
When compressing the Key-Value (KV) cache in Large Language Models (LLMs), researchers observed a fundamental asymmetry. As discovered by Wanyun Cui et al. in Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs, adjacent keys in a sequence are often highly homogeneous (redundant), whereas their corresponding value vectors remain highly heterogeneous (distinct).
To compress the cache without losing information, the authors developed a "cardinality-aware" preservation strategy:
- Sum the Values: Instead of averaging the distinct value vectors (which washes out unique information), they sum them together.
- Track Cardinality: They maintain a cardinality vector tracking the number of merged tokens.
- Adjust the Denominator: During inference, this counter adjusts the attention denominator, ensuring that the aggregated values contribute with mathematically lossless weight to the final output.
A similar philosophy applies when evicting or compacting tokens from long contexts. Adam Zweiger et al. showed in Fast KV Compaction via Attention Matching that a scalar bias term can act as a direct multiplier for attention mass. This prevents the loss of signal that typically occurs when tokens are simply dropped from a cache.
This shift from calculating a "weighted mean" to preserving a "weighted sum" is crucial. As noted by Markus Krimmel et al. in Attention Normalization Impacts Cardinality Generalization in Slot Attention, standard mean normalization strips away information about cluster sizes. Switching to a weighted sum allows latent representations to retain critical scale and importance metrics, vastly improving a model's ability to generalize to scenes with varying numbers of objects.
Generalization Across Modalities and Architectures
The principle of proportional attention weighting has proven to be a universal remedy for the distortions caused by token compression, spanning multiple modalities and model types.
Dense Prediction and Video Transformers
In tasks like object detection or video processing, spatial and temporal boundaries must be preserved precisely. Jin Li et al. designed AiluRus: A Scalable ViT Framework for Dense Prediction to scale the influence of representative tokens based on the number of original tokens they encompass, preventing accuracy loss at object boundaries.
When extending token reduction to the time dimension, Sam Pollard et al. demonstrated in Video, How Do Your Tokens Merge? that adjusting self-attention to account for the spatio-temporal volume of merged patches is vital for maintaining video structure. Furthermore, Xuwei Xu et al. showed in GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation that weighting attention scores by the quantity of original data points prevents the "oversmoothing" of attention maps, keeping representations sharp and discriminative.
Diffusion Transformers (DiTs)
Generative models are highly sensitive to token scaling; improper weighting can manifest as severe visual artifacts. To address this, Elia Peruzzo et al. introduced ADAPTOR: Adaptive Token Reduction for Video Diffusion Transformers, which uses a proportional bias to ensure that tokens representing different spatial scales contribute correctly to the generative process.
For multi-resolution generation, Zeqi Xiao et al. developed Trainable Log-linear Sparse Attention for Efficient Diffusion Transformers, scaling the importance of coarse tokens relative to their receptive field size. This mathematical consistency across compression levels allows a single model to navigate quality-compute trade-offs seamlessly, a property also leveraged by Dong Hoon Lee et al. in Variable-Length Tokenization via Learnable Global Merging for Diffusion Transformers.
Multimodal and Clustering Foundations
When consolidating features via clustering, the representation itself must preserve token mass. Both Joakim Bruslund Haurum et al. (Agglomerative Token Clustering) and Chau Tran et al. (Accelerating Transformers with Spectrum-Preserving Token Merging) proved that calculating representative feature vectors as a weighted average based on cluster size prevents information loss.
This concept also enables efficient multimodal understanding. For instance, Liang Zhang et al. utilized size-weighted attention in TinyChart: Efficient Chart Understanding with Program-of-Thoughts Learning and Visual Token Merging to compress visual regions without losing fine-grained chart details.
Attention Redistribution Without Merging
Even when tokens are not explicitly merged, the principle of preserving the attention budget holds true. Seil Kang et al. identified in See What You Are Told: Visual Attention Sink in Large Multimodal Models that models often waste attention budget on uninformative "sink tokens." By reallocating this wasted attention mass back to informative visual tokens—weighted by their existing importance—they significantly boosted performance without requiring any fine-tuning.
Conclusion
Proportional attention weighting bridges the gap between computational efficiency and mathematical fidelity. By treating token reduction not as a destructive pruning operation, but as a conservation of "attention mass," these methods allow modern transformers to shed redundant computations. Whether through an additive $\log s$ bias in the softmax layer or a cardinality-aware summation of value vectors, preserving the scale of the original data ensures that compressed models remain fast, accurate, and structurally sound.
Key Foundational Work
AiluRus: A Scalable ViT Framework for Dense Prediction (2023) — contributed 1 concept elements to this direction.
Token Merging: Your ViT But Faster (2023) — contributed 1 concept elements to this direction.
Attention Modulation for Zero-Shot Cross-Domain Dialogue State Tracking (2022) — contributed 1 concept elements to this direction.