Abstract
Standard Transformer models rely on the softmax function to compute attention weights, which forces a dense probability distribution where every token receives some non-zero weight. This paper introduces Rectified Linear Attention (ReLA), a sparse alternative that replaces softmax with a simple ReLU activation. ReLA achieves natural sparsity, allowing attention heads to "switch off" when context is irrelevant. Experimental results on multiple WMT translation benchmarks show that ReLA matches the performance of vanilla Transformers while offering superior decoding efficiency and interpretability compared to existing sparse attention methods.
Introduction
In modern Transformer architectures, the softmax function is the standard mechanism for normalizing attention scores. However, softmax has two significant drawbacks: it is inherently dense, assigning weight to even irrelevant noise, and it is computationally rigid, requiring every query to attend to the context with a total mass of exactly 1.0. While previous sparse attention methods like sparsemax or entmax address density, they often introduce complex sorting operations that slow down training and inference. This work explores whether sparsity can be achieved more efficiently using simple linear units.
Rectified Linear Attention (ReLA)
The core proposal is ReLA (Rectified Linear Attention), which replaces the standard softmax pipeline with a ReLU (Rectified Linear Unit) activation:
$$\alpha = \text{ReLU}(f(Q, K^T))$$
Because ReLU outputs zero for any negative input, sparsity emerges naturally without the need for thresholding or sorting. To stabilize training and prevent gradient explosions caused by the removal of softmax normalization, the authors propose two variants:
- ReLA-i: Employs RMSNorm on the attention output combined with a specialized Xavier initialization.
- ReLA-g: Introduces a gating function (a sigmoid-weighted unit) during normalization to provide flexible output scaling.
Unlike softmax, ReLA abandons the probabilistic constraint. A query can attend to context with a total weight greater than one or, uniquely, a total weight of zero—effectively "switching off" the attention head for that token.
Experimental Evaluation
The model was evaluated on five major Machine Translation (MT) benchmarks, including WMT14 (En-De, En-Fr), WMT18 (En-Fi, Zh-En), and WMT16 (Ro-En). These datasets range from 608K to 36M sentence pairs, testing the architecture's scalability across different data regimes.
Key Results and Insights
Performance and Efficiency
ReLA achieves BLEU scores comparable to vanilla Transformers. For instance, on WMT14 En-De, ReLA-g scored 26.6 compared to the baseline's 26.9, while outperforming other sparse variants. Crucially, ReLA maintains $O(n)$ complexity, making it significantly faster than sorting-based sparse methods.
As shown in the decoding speed analysis, ReLA is at least 1.8x faster than sparsemax or entmax, with speeds nearly identical to the vanilla softmax Transformer across various sequence lengths.
Interpretability and Data Sensing
ReLA's sparse weights are more linguistically meaningful than softmax. It achieved an Alignment Error Rate (AER) of 64.46, significantly lower than softmax's 75.95, indicating better alignment with human-annotated word mappings. Furthermore, the authors observed that "null attention" rates (where heads switch off) spike when the model encounters noisy or hallucinated data, suggesting ReLA can serve as an internal gauge for data quality.
Conclusion
Rectified Linear Attention provides a simple, efficient, and interpretable alternative to softmax. By leveraging the natural sparsity of ReLU, it avoids the computational overhead of previous sparse attention techniques while enabling new capabilities like null attention. Future work will explore ReLA's application to extremely long sequences and multi-source architectures where selective ignoring of input sources is critical.
Summary updated