TL;DR
Softmax has long locked Transformer attention weights into strictly non-negative probabilities, forcing the model to rely on static weight matrices to cancel out unwanted context. Cog Attention breaks this convention by allowing dynamically signed attention weights via a simple formulation: $\text{sign}(QK^\top) \odot \text{softmax}(|QK^\top|)$. This enables single attention heads to simultaneously copy and eliminate tokens, alleviates representational over-squashing, and outperforms standard Transformers across model scales up to 1 billion parameters without adding parameters or computational overhead.
What Problem Does This Paper Address?
Standard Transformers constrain attention weights to non-negative values summing to 1. While mathematically convenient, this introduces two major bottlenecks:
- Functional rigidity and "friendly fire": In softmax attention, operations like token deletion (suppressing irrelevant information) must be handled by the static output-value ($OV$) projection matrix. Because attention weights cannot be negative, attending to an irrelevant token to delete it inevitably penalizes neighboring, relevant tokens attended to by the same head.
- Representational collapse ("over-squashing"): In deep networks, earlier tokens repeatedly propagate forward through non-negative mixtures, over-squashing early sequence information into later positions and making representations excessively homogeneous.
Prior attempts to introduce negative weights (such as Differential Attention or Centered Attention) required complex multi-head subtractions, specialized initializations, or heuristic offsets that often hurt training stability.
Methods
The authors introduce Cog Attention, which naturally allows attention weights to take both positive and negative values while preserving training stability:
- Mathematical Formulation: Instead of standard softmax, Cog Attention defines attention row $a_i$ from query-key inner products $p_i = q_i k^\top$ as: $$\phi(p_i)_j = \text{sign}(p_{i,j}) \cdot \text{softmax}(|p_i|)_j$$ Equivalently, it normalizes signed exponentials by the sum of their absolute values: $$\phi(p_i)_j = \frac{\text{sign}(p_{i,j})\exp(|p_{i,j}|)}{\sum_{k} |\text{sign}(p_{i,k})\exp(|p_{i,k})||}$$
- Stability and Overflow Control: To prevent numerical overflow, the maximum absolute value $\max(|p_i|)$ is subtracted before exponentiating. Normalizing by the $L_1$-norm of absolute values ensures the denominator is strictly non-zero, avoiding NaN failures while maintaining unit magnitude: $\sum_j |\phi(p_i)_j| = 1$.
- Architectural Integration: The architecture, dubbed Cogformer, replaces softmax attention in the middle layers while keeping standard softmax in the initial and final layers (which display distinct spectral dynamics). Implemented with a fused Triton kernel, Cog Attention matches the throughput and memory footprint of standard softmax attention.
Datasets
- Pre-training: Evaluated on models trained from scratch on 100 billion tokens of the open-source RedPajama dataset.
- Downstream Benchmarks:
- Reasoning & Commonsense: ARC-Easy, ARC-Challenge, PIQA, SIQA.
- General NLP & NLI (1-shot): SST-2 (sentiment), MRPC (paraphrase), MNLI (inference).
- Interpretability & Diagnostic Tasks:
- Indirect Object Identification (IOI): 100-sample mechanistic benchmark evaluating token elimination circuits.
- Context Distinction (Synthetic): Two custom synthetic tasks—Finding a Zero and Counting Ones—measuring representation divergence as sequence length $n$ increases up to 2,000 tokens.
Key Insights and Ideas
- Dynamic elimination vs. static elimination: In IOI circuit analyses, standard attention heads require an $OV$ projection matrix with strongly negative eigenvalues ($-0.95$) to eliminate subjects ($S$), inadvertently suppressing target indirect objects ($IO$, correlation $-0.21$). Cog Attention dynamically flips the sign in the QK inner product, eliminating $S$ via negative weights while leaving $IO$ intact (correlation $+0.13$), freeing the $OV$ matrix to perform semantic refinement.
- Mitigating over-squashing: By assigning negative weights to earlier, redundant tokens, Cog Attention dampens the cumulative information funnel into final positions. In synthetic stress tests, Cogformer maintains significantly higher distinguishability ($\Delta L_{+\infty}$) at long sequence lengths than vanilla Transformers.
- Elimination of massive attention sinks: Inspection of layer activations reveals that Cogformer heads display far fewer "attention sinks" (massive attention dumped onto initial tokens) compared to vanilla Transformers, suggesting more heads remain actively engaged in context processing.
Key Results
- 141M Scale: Cogformer achieves 44.35% average accuracy across 7 NLP benchmarks, outperforming standard Transformer (42.98%), Differential Attention (43.06%), and Centered Attention (42.96%).
- Scaling to 600M & 1B:
- At 600M, Cogformer leads vanilla Transformers with 45.45% vs. 44.60% average accuracy.
- At 1B, Cogformer maintains the advantage with 48.01% vs. 46.99% average accuracy, posting consistent gains on MRPC (66.42% vs. 63.73%) and MNLI (55.16% vs. 53.67%).
- Efficiency: Fused Triton implementations achieve identical throughput and memory scaling to standard Flash-style softmax attention across context lengths up to 12,000 tokens.
Limitations and Future Work
- Hybrid layer requirement: The authors note that Cogformer still requires standard softmax attention in the first and last few layers to maintain optimal performance and stability.
- Direct evidence for reduced redundancy: While visual attention patterns show less attention-sink behavior and more varied representations, direct mathematical metrics quantifying reduced parameter redundancy remain an open question.
- Future scope: Testing whether reduced attention sinks directly translate into practical wins for KV-cache compression, long-context retrieval (RAG), and context length extrapolation.
Summary updated