Abstract
Standard Transformer models rely on the softmax function for attention, which constrains weights to a non-negative probability distribution. While effective, this limitation forces attention heads to use positive weights for both information retrieval and suppression, often leading to "friendly fire" where useful tokens are inadvertently diminished. This paper introduces Cog Attention, a novel mechanism that enables negative attention weights through a new SignExp function. By allowing tokens to be explicitly "deleted" or "canceled out," Cog Attention enhances model expressiveness and mitigates representational collapse. Our results show that Cogformer models consistently outperform vanilla Transformers in language modeling and image generation tasks with no additional computational overhead.
Introduction
Since the introduction of the Transformer, the softmax function has been the standard for calculating attention weights. By design, softmax produces non-negative values that sum to one. However, this non-negativity constraint fundamentally limits the "expressiveness" of the model. In a standard attention head, the Query-Key (QK) product identifies which tokens to attend to, while the Output-Value (OV) matrix determines the transformation (e.g., copying or deleting).
Because weights are strictly positive, a head attempting to "delete" an irrelevant token often partially suppresses useful tokens—a phenomenon termed "friendly fire." Furthermore, in deep models, earlier tokens tend to get "over-squashed" into later positions, leading to representational collapse, where hidden states become indistinguishable. Cog Attention addresses these issues by breaking the non-negativity barrier.
Methods: The Cog Attention Mechanism
The core of Cog Attention is the replacement of the standard softmax with the SignExp function. This approach shifts the decision to "copy" or "delete" from the static OV matrix to the dynamic QK inner product.
The mechanism consists of three primary components:
- Sign Preservation: Instead of squashing all values into a positive range, Cog Attention calculates the exponential of the absolute inner product but reapplies the original sign of the QK product. This allows the model to learn positive signs for "copying" and negative signs for "elimination."
- Absolute Normalization: To maintain training stability, weights are normalized by the sum of their absolute values. This ensures the total influence remains bounded while permitting individual negative weights.
- Numerical Stability: A modified "max-subtraction" trick (using the maximum absolute value) is employed to prevent exponential overflow, ensuring the implementation is as stable as standard softmax.
The resulting architecture, Cogformer, serves as a drop-in replacement for vanilla Transformers. To optimize convergence, the authors found that retaining standard softmax in the first and last layers while using Cog Attention in the middle layers provides the best balance of speed and expressiveness.
Key Insights and Results
Cog Attention introduces a functional shift in how information is processed. By allowing negative weights, the model can "cancel out" redundant information pathways, preventing early tokens from being forced into every subsequent representation.
Performance Across Benchmarks
The authors evaluated Cogformer across several domains, demonstrating its scalability and efficiency:
- Language Modeling: A 141M parameter Cogformer achieved an average accuracy of 44.35% across NLP tasks (ARC, PIQA, etc.), outperforming the vanilla Transformer (42.98%). These gains persisted as models scaled to 600M and 1B parameters.
- Image Generation: In U-ViT diffusion models, Cog Attention improved image quality, achieving lower FID scores on CIFAR-10 (3.27 vs. 3.39) and MS-COCO (5.85 vs. 5.99).
- Efficiency: Custom Triton kernels ensure that Cog Attention introduces zero additional time overhead and negligible memory increase compared to standard softmax.
Analysis of attention patterns reveals that Cogformer relies less on "attention sinks" (e.g., [CLS] or punctuation tokens). Instead, more heads are actively engaged in processing meaningful input, leading to the characteristic "cog-like" patterns that give the method its name.
Conclusion
Cog Attention represents a significant step toward more expressive Transformer architectures. By enabling negative weights, it provides a principled way to mitigate representational collapse and improve information flow. While the current implementation benefits from hybrid convergence strategies, the expressive power of the processing layers is greatly enhanced. Future work will investigate the impact of negative weights on long-context tasks and potential parameter redundancy reductions.
Summary updated