Breaking the Convex Hull Constraint in Attention

18 concepts11 papers1 proposal

At the core of the modern Transformer is the softmax-based attention mechanism. While remarkably successful, standard attention suffers from a fundamental geometric limitation: it acts as a convex aggregator. Because the softmax function outputs weights that are strictly non-negative ($\alpha_i \ge 0$) and sum to one ($\sum \alpha_i = 1$), the resulting output vector is mathematically confined to the convex hull of the input value vectors.

Intuitively, if you imagine the input embeddings as points in a high-dimensional space, standard attention can only generate new points inside the boundary formed by stretching a rubber band around those inputs. It can blend existing features, but it cannot extrapolate beyond them, nor can it actively cancel out or subtract specific components. A growing body of research focuses on breaking this constraint by introducing subtractive attention and negative weights, unlocking a new dimension of expressiveness for deep learning models.

The Cost of Positivity: Noise and Bounded Representations

The non-negativity constraint of standard attention introduces two primary limitations: an inability to suppress irrelevant information and a strict representation ceiling.

1. The Inability to Suppress Noise

In language and reasoning, understanding what something is not is often as important as understanding what it is. However, as Zhengxuan Wu et al. observe in Context-Guided BERT for Targeted Aspect-Based Sentiment Analysis, standard attention can at best assign a weight of zero to a token. It cannot actively "de-attend" to or discount misleading words. For instance, a generally positive word might act as a negative indicator in a highly specific context, but standard attention lacks the mechanism to subtract its influence.

This limitation also manifests as "attention noise." As explained by Chaerin Kong et al. in Understanding Differential Transformer Unchains Pretrained Self-Attentions, standard softmax forces a model to allocate attention across background noise, whereas an ideal system would explicitly model negative relevance to cancel out irrelevant background signals.

2. The Representation Ceiling

When a model is forced to operate purely within a convex hull, it cannot synthesize extreme or novel feature representations that lie outside the span of its immediate inputs. Jingquan Yan et al. identify this in Breaking the Correlation Plateau: On the Optimization and Capacity Limits of Attention-Based Regressors as the root cause of the "Pearson Correlation Coefficient Plateau" in regression tasks. When processing highly homogeneous data, a convex aggregator cannot extrapolate to predict extreme values or capture sharp contrastive trends because the output is mathematically bounded by the range of the input values.

Similarly, Hareesh Bahuleyan et al. argue in Variational Attention for Sequence-to-Sequence Models that this rigid geometric boundary prevents deterministic attention models from exploring the surrounding representation space, severely limiting the diversity and creativity of generated text.

Standard Attention (Convex Hull):
Input Vectors:  [ * ]     [ * ]     [ * ]
Output Vector:         [ X ]  <-- Trapped inside the boundary

Subtractive Attention (Unconstrained):
Input Vectors:  [ * ]     [ * ]     [ * ]
Output Vector:                             [ X ] <-- Can extrapolate outside

Mechanisms for Subtractive and Negative Attention

To bypass the convex hull bottleneck, researchers have proposed several architectural innovations that allow attention mechanisms to perform subtractive operations.

Direct Negative Weighting

The most direct way to break the constraint is to allow the attention weights themselves to be negative. Ang Lv et al. introduce this capability in More Expressive Attention with Negative Weights via "Cog Attention." By allowing negative weights, a single attention head can simultaneously perform multiple distinct operations—such as reinforcing relevant tokens while deleting or refining others—preventing representational collapse.

In the domain of efficient architectures, Jiecheng Lu et al. propose ZeroS: Zero‑Sum Linear Attention for Efficient Transformers. By removing the constant averaging term typical of linear softmax approximations, the authors enable contrastive operations and negative weights within a single layer, achieving linear $O(N)$ time complexity without sacrificing expressive power.

Dual-Affinity and Differential Schemes

Rather than modifying the softmax function directly, some approaches achieve subtraction by combining multiple attention maps. In Understanding Differential Transformer Unchains Pretrained Self-Attentions, the authors leverage a differential attention mechanism that subtracts one attention map from another. This explicit subtraction acts as a high-pass filter, canceling out common-mode noise and sharpening the model's focus on core information.

Taking a compositional approach, Yi Tay et al. develop Compositional De-Attention Networks (CoDA). CoDA replaces standard softmax with a dual-affinity framework that explicitly models both similarity and dissimilarity. This allows the network to dynamically choose whether to add, subtract, or entirely nullify input vectors during token interaction.

Alternative Mixing and Subspace Filtering

Other methods replace the dot-product softmax paradigm entirely to avoid the convex hull. Tong Yu et al. propose Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attention, which factorizes the token-mixing matrix into sparse, full-rank components parameterized by multi-layer perceptrons (MLPs). This completely unchains the mixing process from the convex hull constraint.

From a graph signal processing perspective, Zichen Miao et al. introduce Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models. The authors demonstrate that standard attention acts as a restricted non-negative graph filter. By learning a set of unconstrained coefficients to mix attention heads, they expand the filter subspace to allow general convolutional operations, including subtraction.

Why Subtractive Attention Works: Theoretical and Practical Impact

Allowing models to subtract information yields measurable improvements across theoretical capacity, interpretability, and task performance.

Theoretical Superiority

The mathematical necessity of negative weights is rigorously demonstrated by Yingqian Cui et al. in Superiority of Multi-Head Attention: A Theoretical Study in Shallow Transformers in In-Context Linear Regression. When analyzing how Transformers perform in-context learning for linear regression, the authors proved that single-head attention is fundamentally limited by the non-negativity of softmax. Multi-head attention achieves superior flexibility precisely because the combination of multiple heads allows the model to effectively simulate "negative" attention weights, a capability required for precise function approximation and data extrapolation.

Aligning Attention with True Feature Importance

The constraint of non-negativity also creates a disconnect between raw attention weights and post-hoc interpretability metrics. In Attention Meets Post-hoc Interpretability: A Mathematical Perspective, Gianluigi Lopardo et al. show that post-hoc explanation methods (like LIME or gradient-based saliency maps) naturally capture directional, negative contributions to a prediction. Because raw attention weights are forced to be positive, they are mathematically incapable of identifying tokens that act as counter-evidence. Subtractive attention mechanisms bridge this gap, allowing the raw attention weights to faithfully reflect both positive and negative evidence.

Application Benefits

In practice, the ability to suppress irrelevant tokens leads to sharper boundaries and cleaner representations in downstream tasks:

By moving beyond simple additive blending, subtractive attention transforms the attention mechanism from a passive convex aggregator into a fully flexible vector space operator capable of addition, subtraction, and extrapolation.

Key Foundational Work

Compositional De-Attention Networks (2019) — contributed 1 concept elements to this direction.
Variational Attention for Sequence-to-Sequence Models (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subtractive_attention_9d344fe2,
  title = {subtractive attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subtractive-attention/rdr_9d344fe2794047e38c104603fb1d3e2e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_subtractive_attention_9d344fe2_v1,
  title = {subtractive attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subtractive-attention/rdr_9d344fe2794047e38c104603fb1d3e2e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake