More Expressive Attention with Negative Weights

Ang Lv, Ruobing Xie, Shuaipeng Li, Jiayi Liao, Xingwu Sun, Zhanhui Kang, Di Wang, Rui YanSubmission to ICLR 2026 · Withdrawn

TL;DR

Softmax has long locked Transformer attention weights into strictly non-negative probabilities, forcing the model to rely on static weight matrices to cancel out unwanted context. Cog Attention breaks this convention by allowing dynamically signed attention weights via a simple formulation: $\text{sign}(QK^\top) \odot \text{softmax}(|QK^\top|)$. This enables single attention heads to simultaneously copy and eliminate tokens, alleviates representational over-squashing, and outperforms standard Transformers across model scales up to 1 billion parameters without adding parameters or computational overhead.

What Problem Does This Paper Address?

Standard Transformers constrain attention weights to non-negative values summing to 1. While mathematically convenient, this introduces two major bottlenecks:

  1. Functional rigidity and "friendly fire": In softmax attention, operations like token deletion (suppressing irrelevant information) must be handled by the static output-value ($OV$) projection matrix. Because attention weights cannot be negative, attending to an irrelevant token to delete it inevitably penalizes neighboring, relevant tokens attended to by the same head.
  2. Representational collapse ("over-squashing"): In deep networks, earlier tokens repeatedly propagate forward through non-negative mixtures, over-squashing early sequence information into later positions and making representations excessively homogeneous.

Prior attempts to introduce negative weights (such as Differential Attention or Centered Attention) required complex multi-head subtractions, specialized initializations, or heuristic offsets that often hurt training stability.

Methods

The authors introduce Cog Attention, which naturally allows attention weights to take both positive and negative values while preserving training stability:

  • Mathematical Formulation: Instead of standard softmax, Cog Attention defines attention row $a_i$ from query-key inner products $p_i = q_i k^\top$ as: $$\phi(p_i)_j = \text{sign}(p_{i,j}) \cdot \text{softmax}(|p_i|)_j$$ Equivalently, it normalizes signed exponentials by the sum of their absolute values: $$\phi(p_i)_j = \frac{\text{sign}(p_{i,j})\exp(|p_{i,j}|)}{\sum_{k} |\text{sign}(p_{i,k})\exp(|p_{i,k})||}$$
  • Stability and Overflow Control: To prevent numerical overflow, the maximum absolute value $\max(|p_i|)$ is subtracted before exponentiating. Normalizing by the $L_1$-norm of absolute values ensures the denominator is strictly non-zero, avoiding NaN failures while maintaining unit magnitude: $\sum_j |\phi(p_i)_j| = 1$.
  • Architectural Integration: The architecture, dubbed Cogformer, replaces softmax attention in the middle layers while keeping standard softmax in the initial and final layers (which display distinct spectral dynamics). Implemented with a fused Triton kernel, Cog Attention matches the throughput and memory footprint of standard softmax attention.

Datasets

  • Pre-training: Evaluated on models trained from scratch on 100 billion tokens of the open-source RedPajama dataset.
  • Downstream Benchmarks:
  • Reasoning & Commonsense: ARC-Easy, ARC-Challenge, PIQA, SIQA.
  • General NLP & NLI (1-shot): SST-2 (sentiment), MRPC (paraphrase), MNLI (inference).
  • Interpretability & Diagnostic Tasks:
  • Indirect Object Identification (IOI): 100-sample mechanistic benchmark evaluating token elimination circuits.
  • Context Distinction (Synthetic): Two custom synthetic tasks—Finding a Zero and Counting Ones—measuring representation divergence as sequence length $n$ increases up to 2,000 tokens.

Key Insights and Ideas

  • Dynamic elimination vs. static elimination: In IOI circuit analyses, standard attention heads require an $OV$ projection matrix with strongly negative eigenvalues ($-0.95$) to eliminate subjects ($S$), inadvertently suppressing target indirect objects ($IO$, correlation $-0.21$). Cog Attention dynamically flips the sign in the QK inner product, eliminating $S$ via negative weights while leaving $IO$ intact (correlation $+0.13$), freeing the $OV$ matrix to perform semantic refinement.
  • Mitigating over-squashing: By assigning negative weights to earlier, redundant tokens, Cog Attention dampens the cumulative information funnel into final positions. In synthetic stress tests, Cogformer maintains significantly higher distinguishability ($\Delta L_{+\infty}$) at long sequence lengths than vanilla Transformers.
  • Elimination of massive attention sinks: Inspection of layer activations reveals that Cogformer heads display far fewer "attention sinks" (massive attention dumped onto initial tokens) compared to vanilla Transformers, suggesting more heads remain actively engaged in context processing.

Key Results

  • 141M Scale: Cogformer achieves 44.35% average accuracy across 7 NLP benchmarks, outperforming standard Transformer (42.98%), Differential Attention (43.06%), and Centered Attention (42.96%).
  • Scaling to 600M & 1B:
  • At 600M, Cogformer leads vanilla Transformers with 45.45% vs. 44.60% average accuracy.
  • At 1B, Cogformer maintains the advantage with 48.01% vs. 46.99% average accuracy, posting consistent gains on MRPC (66.42% vs. 63.73%) and MNLI (55.16% vs. 53.67%).
  • Efficiency: Fused Triton implementations achieve identical throughput and memory scaling to standard Flash-style softmax attention across context lengths up to 12,000 tokens.

Limitations and Future Work

  • Hybrid layer requirement: The authors note that Cogformer still requires standard softmax attention in the first and last few layers to maintain optimal performance and stability.
  • Direct evidence for reduced redundancy: While visual attention patterns show less attention-sink behavior and more varied representations, direct mathematical metrics quantifying reduced parameter redundancy remain an open question.
  • Future scope: Testing whether reduced attention sinks directly translate into practical wins for KV-cache compression, long-context retrieval (RAG), and context length extrapolation.

Summary updated

Abstract

We propose a novel attention mechanism, named Cog Attention, that enables attention weights to be negative for enhanced expressiveness. This stems from two key factors: (1) Cog Attention enhances parameter flexibility. For example, unlike traditional softmax attention heads, which use a static output-value (OV) matrix to delete or copy inputs that the heads attend to, Cog Attention naturally learns to use the sign of dynamic query-key (QK) inner products to represent these operations. This enables Cog Attention to perform multiple operations simultaneously within a single head. Meanwhile, Cog Attention’s OV matrix can focus more on refinement. (2) Cog Attention improves the model’s robustness against representational collapse by preventing earlier tokens from "over-squashing" into later positions. We develop Transformer-like models that use Cog Attention as attention modules, including decoder-only models with up to 3 billion parameters for language modeling. Experiments show that models using Cog Attention exhibit superior performance compared to those employing traditional softmax attention modules. Our approach suggests a promising research direction for rethinking and breaking the entrenched constraints of traditional softmax attention, such as the requirement for non-negative weights.
BibTeX
@misc{lv2024more,
  author = {Ang Lv and Ruobing Xie and Shuaipeng Li and Jiayi Liao and Xingwu Sun and Zhanhui Kang and Di Wang and Rui Yan},
  title = {{More Expressive Attention with Negative Weights}},
  year = {2024},
  eprint = {2411.07176},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.07176},
}
Figure