Transformers are computationally expensive because their attention mechanism scales quadratically with the number of input tokens. To solve this, a growing body of research focuses on Token Merging: identifying redundant patches of an image or segments of text and fusing them into a single representative token.
However, simply averaging tokens creates a mathematical "mass" problem. If ten background tokens are merged into one, that new token should theoretically exert ten times the influence on the rest of the model. Without adjustment, the model treats the merged token as a single entity, effectively "silencing" the importance of the compressed region. Proportional Attention Weighting is the mechanism used to restore this mathematical integrity, ensuring that compressed models behave like their full-resolution counterparts.
The Problem of Vanishing Token Mass
In a standard attention layer, every token (or "key") competes for a slice of the attention probability via the Softmax operation. When Daniel Bolya and Judy Hoffman introduced Token Merging for Fast Stable Diffusion, they highlighted a fundamental discrepancy: if you merge $r$ tokens into one, the resulting token still only receives one "vote" in the Softmax denominator.
This leads to a loss of global context. For example, in a vision task, if a large but redundant sky region is compressed into a single token, the model might under-prioritize the sky's influence on the lighting of the foreground. As noted in Co-Me: Confidence-Guided Token Merging for Visual Geometric Transformers, this distortion prevents the model from maintaining an accurate global scene representation.
Correcting the Softmax with Logarithmic Bias
The most common way to fix this is to inject the "count" of original tokens back into the attention calculation. If a merged token represents $s$ original patches, we want its attention weight to be scaled by $s$.
Mathematically, this is often achieved by adding a bias term to the attention logits (the values before the Softmax). As implemented in Fast Autoregressive Video Diffusion and World Models and BccT: an efficient transformer model for blood cell classification, adding $\log(s)$ to the logit of a merged token ensures that its post-Softmax weight is proportional to its original size.
This "log-linear" correction allows the model to remain "mass-conservative." Yuheng Qiu and colleagues demonstrate that this adjustment is critical for high-fidelity tasks like 3D reconstruction, where losing the relative importance of spatial areas would lead to geometric artifacts.
Cardinality-Aware KV Caching
This concept extends beyond vision into Large Language Models (LLMs). In Homogeneous Keys, Heterogeneous Values, Wanyun Cui and Mingwei Xu observe that while Keys in a KV cache are often redundant, the Values are not.
To compress the cache without losing information, they use AsymKV, which merges Keys but keeps a "cardinality vector"—a counter of how many original tokens each merged Key represents. During inference, this counter adjusts the attention denominator. This ensures that even when the cache is heavily compressed, the aggregated Values contribute to the final output with the correct mathematical weight, allowing for nearly lossless long-context retrieval.
Similarly, Adam Zweiger and the team at MIT developed Fast KV Compaction via Attention Matching, which uses a scalar bias term as a multiplier for "attention mass." This prevents the loss of signal that typically occurs when tokens are simply evicted from the cache.
Maintaining Fidelity Across Scales
In generative models, proportional weighting is the difference between a fast model and a fast accurate model. ADAPTOR, a framework for video diffusion by Elia Peruzzo and others, uses this weighting to balance high-motion areas with compressed backgrounds. By scaling the influence of coarse tokens relative to their receptive field size, the model ensures that fine-grained details and broad background strokes are integrated accurately.
This theme is echoed in AiluRus, where Jin Li scales the influence of representative tokens to maintain the integrity of dense prediction tasks like object detection. Whether it is GTP-ViT propagating information from discarded tokens to their neighbors or TinyChart weighting visual tokens to preserve chart data, the core intuition remains the same:
Efficiency should not come at the cost of representation. By treating tokens as "mass" rather than just "points," proportional attention weighting allows Transformers to shrink in size while maintaining their original mathematical footprint.