More Expressive Attention with Negative Weights

Ang Lv, Ruobing Xie, Shuaipeng Li, Jiayi Liao, Xingwu Sun, Zhanhui Kang, Di Wang, Rui Yan

Abstract

Standard Transformer models rely on the softmax function for attention, which constrains weights to a non-negative probability distribution. While effective, this limitation forces attention heads to use positive weights for both information retrieval and suppression, often leading to "friendly fire" where useful tokens are inadvertently diminished. This paper introduces Cog Attention, a novel mechanism that enables negative attention weights through a new SignExp function. By allowing tokens to be explicitly "deleted" or "canceled out," Cog Attention enhances model expressiveness and mitigates representational collapse. Our results show that Cogformer models consistently outperform vanilla Transformers in language modeling and image generation tasks with no additional computational overhead.

Introduction

Since the introduction of the Transformer, the softmax function has been the standard for calculating attention weights. By design, softmax produces non-negative values that sum to one. However, this non-negativity constraint fundamentally limits the "expressiveness" of the model. In a standard attention head, the Query-Key (QK) product identifies which tokens to attend to, while the Output-Value (OV) matrix determines the transformation (e.g., copying or deleting).

Because weights are strictly positive, a head attempting to "delete" an irrelevant token often partially suppresses useful tokens—a phenomenon termed "friendly fire." Furthermore, in deep models, earlier tokens tend to get "over-squashed" into later positions, leading to representational collapse, where hidden states become indistinguishable. Cog Attention addresses these issues by breaking the non-negativity barrier.

Methods: The Cog Attention Mechanism

The core of Cog Attention is the replacement of the standard softmax with the SignExp function. This approach shifts the decision to "copy" or "delete" from the static OV matrix to the dynamic QK inner product.

The SignExp function preserves the sign of the inner product while applying exponential scaling, allowing for both positive (copying) and negative (deletion) attention weights.

The mechanism consists of three primary components:

  1. Sign Preservation: Instead of squashing all values into a positive range, Cog Attention calculates the exponential of the absolute inner product but reapplies the original sign of the QK product. This allows the model to learn positive signs for "copying" and negative signs for "elimination."
  2. Absolute Normalization: To maintain training stability, weights are normalized by the sum of their absolute values. This ensures the total influence remains bounded while permitting individual negative weights.
  3. Numerical Stability: A modified "max-subtraction" trick (using the maximum absolute value) is employed to prevent exponential overflow, ensuring the implementation is as stable as standard softmax.

The resulting architecture, Cogformer, serves as a drop-in replacement for vanilla Transformers. To optimize convergence, the authors found that retaining standard softmax in the first and last layers while using Cog Attention in the middle layers provides the best balance of speed and expressiveness.

Key Insights and Results

Cog Attention introduces a functional shift in how information is processed. By allowing negative weights, the model can "cancel out" redundant information pathways, preventing early tokens from being forced into every subsequent representation.

Comparison of Softmax and Cog Attention weights. (a-b) Heatmaps show Cog Attention utilizing negative weights (red) for suppression. (c-d) Correlation analysis demonstrates how Cog Attention better distinguishes between Subject (S) and Indirect Object (IO) tokens.

Performance Across Benchmarks

The authors evaluated Cogformer across several domains, demonstrating its scalability and efficiency:

  • Language Modeling: A 141M parameter Cogformer achieved an average accuracy of 44.35% across NLP tasks (ARC, PIQA, etc.), outperforming the vanilla Transformer (42.98%). These gains persisted as models scaled to 600M and 1B parameters.
  • Image Generation: In U-ViT diffusion models, Cog Attention improved image quality, achieving lower FID scores on CIFAR-10 (3.27 vs. 3.39) and MS-COCO (5.85 vs. 5.99).
  • Efficiency: Custom Triton kernels ensure that Cog Attention introduces zero additional time overhead and negligible memory increase compared to standard softmax.

Analysis of attention patterns reveals that Cogformer relies less on "attention sinks" (e.g., [CLS] or punctuation tokens). Instead, more heads are actively engaged in processing meaningful input, leading to the characteristic "cog-like" patterns that give the method its name.

Conclusion

Cog Attention represents a significant step toward more expressive Transformer architectures. By enabling negative weights, it provides a principled way to mitigate representational collapse and improve information flow. While the current implementation benefits from hybrid convergence strategies, the expressive power of the processing layers is greatly enhanced. Future work will investigate the impact of negative weights on long-context tasks and potential parameter redundancy reductions.

Summary updated

Abstract

We propose a novel attention mechanism, named Cog Attention, that enables attention weights to be negative for enhanced expressiveness, which stems from two key factors: (1) Cog Attention enhances parameter flexibility. For example, unlike traditional softmax attention heads that use a static output-value (OV) matrix to delete or copy inputs that the heads attend to, Cog Attention naturally learns to use the sign of dynamic query-key (QK) inner products to represent these operations. This enables Cog Attention to perform multiple operations simultaneously within a single head. Meanwhile, Cog Attention's OV matrix can focus more on refinement or modification. (2) Cog Attention enhances the model's robustness against representational collapse by preventing the ``over-squashing'' of earlier tokens into later positions. We develop Transformer-like models which use Cog Attention as attention modules, including decoder-only models at various scales for language modeling and U-ViT diffusion models for image generation. Experiments show that models using Cog Attention exhibit superior performance compared to those employing traditional softmax attention modules. Our approach suggests a promising research direction for rethinking and breaking the entrenched constraints of traditional softmax attention, such as the requirement for non-negative weights.
BibTeX
@misc{lv2024more,
  author = {Ang Lv and Ruobing Xie and Shuaipeng Li and Jiayi Liao and Xingwu Sun and Zhanhui Kang and Di Wang and Rui Yan},
  title = {{More Expressive Attention with Negative Weights}},
  year = {2024},
  eprint = {2411.07176},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.07176},
}
Figure