Abstract
Compositional De-Attention (CoDA) moves beyond standard "weighted sum" attention by allowing models to add, subtract, or nullify input vectors. By replacing the traditional Softmax operator with a dual-affinity scheme that captures both similarity and dissimilarity, CoDA achieves state-of-the-art results across a wide range of NLP tasks, from machine translation to mathematical reasoning.
What Problem Does This Paper Address?
In the world of deep learning, "attention" is the gold standard for processing sequences. However, standard attention mechanisms (like those found in Transformers) are fundamentally limited by their reliance on the Softmax operator. Softmax forces the model to perform a weighted average—a convex combination—of input tokens, meaning all attention weights must sum to one.
The authors argue that this "re-weighting" is too restrictive. In natural language and logic, we do not just weigh inputs; we often need to negate information, subtract specific features, or completely ignore irrelevant noise. Standard attention cannot easily "subtract" a vector. CoDA addresses this limitation by proposing a "quasi-attention" mechanism that treats representation learning as a compositional process, giving the model the agency to decide whether to include, flip, or delete information.
Methods: The Dual Affinity Scheme
The core innovation of CoDA is the Dual Affinity Scheme. Instead of computing a single similarity matrix, the model computes two distinct matrices:
- Similarity ($E$): Typically a dot product between vectors, capturing how "aligned" two tokens are.
- Dissimilarity ($N$): Computed using a distance metric (like negative L1 distance), capturing how "different" tokens are.
These two matrices are combined into a final quasi-attention matrix $M$ using the following logic:
$$M = \tanh(E) \odot \text{sigmoid}(N)$$
- The $\tanh(E)$ term allows for values between -1 and 1. This enables the model to either add (+1) or subtract (-1) a vector, rather than just scaling it positively.
- The $\text{sigmoid}(N)$ term acts as a gate. If dissimilarity is high, the gate closes toward 0, effectively deleting (nullifying) the token's contribution.
By removing the Softmax, the model is no longer forced to make all weights sum to one, allowing for much greater flexibility and expressiveness in how it composes representations.
Key Insights and Ideas
- Softmax-less Attention is Viable: One of the most surprising takeaways is that the Softmax operator—long considered essential for stability and normalization in attention—can be successfully replaced with gated $\tanh$ and $\text{sigmoid}$ functions without losing performance.
- Subtraction as an Inductive Bias: By explicitly allowing the model to "subtract" vectors, CoDA provides a powerful tool for tasks requiring logic or contradiction (like Natural Language Inference or Mathematics).
- Dissimilarity as "De-attention": Using a distance metric to "erase" unnecessary pairwise scores provides a more robust gating mechanism than similarity alone, protecting the model from over-relying on raw alignment.
- Universal Applicability: CoDA is not a standalone model; it is a "neural component" that can be dropped into existing architectures like the Transformer or Decomposable Attention networks.
Datasets and Key Results
The authors evaluated CoDA across an extensive suite of benchmarks, including Open Domain QA (SearchQA, Quasar-T), Retrieval/Ranking (TrecQA, WikiQA, UDC), Natural Language Inference (SNLI, MNLI, SciTail, DNLI), Machine Translation, Sentiment Analysis, and Text2Code Generation.
CoDA consistently outperformed vanilla attention mechanisms across the board:
- Machine Translation: The CoDA Transformer improved upon the base Transformer by +1.4 BLEU points on the IWSLT'15 English-Vietnamese dataset.
- Mathematical Reasoning: On the MLU dataset (a character-level transduction task involving arithmetic operations), the CoDA Transformer achieved 84.3% accuracy, a massive +8.2% jump over the base Transformer, nearly solving the complex arithmetic task.
- Natural Language Inference (NLI): Achieved a new state-of-the-art on the Dialogue NLI (DNLI) leaderboard with 88.8% accuracy.
- Text2Code: Reached 97.7% accuracy on AlgoLisp, outperforming the standard Transformer (96.8%).
- Open Domain QA: Improved the DecaProp model by +2.7% F1 on the Quasar-T dataset.
Limitations and Future Work
While CoDA shows impressive gains, the paper notes a few areas for further exploration:
- Hyperparameter Sensitivity: The scaling constants ($\alpha$ and $\beta$) act as "temperatures" for the $\tanh$ and $\text{sigmoid}$ functions. Finding the right balance for these is crucial to prevent the matrices from saturating too early during training.
- Centering Requirements: To ensure the "subtraction" and "gating" logic works effectively, the authors often had to center the affinity matrices to zero mean. This adds a small amount of computational overhead and architectural complexity.
- Scaling to Vision: The experiments were strictly limited to NLP. It remains to be seen if the "add/subtract/delete" intuition translates as effectively to computer vision tasks, such as in Vision Transformers.
Summary updated