Abstract: Recently, it has been argued that encoder-decoder models can be made more interpretable by replacing the softmax function in the attention with its sparse variants. In this work, we introduce a novel, simple method for achieving sparsity in attention: we replace the softmax activation with a ReLU, and show that sparsity naturally emerges from such a formulation. Training stability is achieved with layer normalization with either a specialized initialization or an additional gating function. Our model, which we call Rectified Linear Attention (ReLA), is easy to implement and more efficient than previously proposed sparse attention mechanisms. We apply ReLA to the Transformer and conduct experiments on five machine translation tasks. ReLA achieves translation performance comparable to several strong baselines, with training and decoding speed similar to that of the vanilla attention. Our analysis shows that ReLA delivers high sparsity rate and head diversity, and the induced cross attention achieves better accuracy with respect to source-target word alignment than recent sparsified softmax-based models. Intriguingly, ReLA heads also learn to attend to nothing (i.e. 'switch off') for some queries, which is not possible with sparsified softmax alternatives.
Introduction
In Transformer-based models, the softmax function is the standard mechanism for calculating attention weights. Despite its ubiquity, softmax possesses two significant limitations. First, it is inherently dense, assigning non-zero weights to every token in a sequence regardless of relevance. This density complicates model interpretability and increases computational overhead. Second, softmax enforces a "fixed mass" constraint where attention weights must sum to one. This forces the model to attend to some part of the context even when the query is irrelevant to the available information.
While sparse alternatives like sparsemax and entmax have been proposed to address density, they rely on computationally expensive sorting operations. This work introduces Rectified Linear Attention (ReLA), a simpler approach that achieves sparsity without the performance penalties of previous methods.
Rectified Linear Attention (ReLA)
The core innovation of ReLA is the replacement of the softmax activation with a Rectified Linear Unit (ReLU). Because ReLU outputs zero for any negative input, sparsity emerges naturally from the attention scores.
To ensure training stability, which can be compromised by the unbounded nature of ReLU compared to softmax, the authors introduce two key modifications:
- Normalization: RMSNorm (Root Mean Square Layer Normalization) is applied to the attention outputs to maintain activations within a stable range.
- Stabilization Variants: Two versions were developed to manage attention gain: ReLA-i, which utilizes a specialized Xavier initialization for normalization parameters, and ReLA-g, which incorporates a sigmoid-weighted gating function.
These modifications allow ReLA to serve as a drop-in replacement for standard attention, requiring no custom inference kernels or complex sorting.
Experimental Evaluation
The effectiveness of ReLA was evaluated across five machine translation benchmarks from the WMT (Workshop on Machine Translation) series: En-De, En-Fr, En-Fi, Zh-En, and Ro-En. These datasets vary significantly in scale, from 600,000 to 36 million training pairs, providing a comprehensive test of the model's robustness across different language pairs and data scales.
Results and Analysis
ReLA achieves translation quality comparable to vanilla Transformers while offering superior efficiency and interpretability features.
Translation Performance and Efficiency
On the WMT14 En-De task, ReLA-g achieved a BLEU score of 26.6, nearly matching the baseline Transformer's 26.9. Crucially, ReLA avoids the speed bottlenecks associated with other sparse attention mechanisms.
As shown in the figure above, ReLA-g maintains a decoding speed of approximately 0.98x relative to vanilla softmax. In contrast, sparsemax and entmax drop to roughly 0.54x and 0.49x, respectively. This efficiency makes ReLA the first sparse attention mechanism to achieve near-parity with standard Transformers in terms of throughput.
Sparsity and "Null Attention"
A unique property of ReLA is its ability to produce "null attention." Unlike softmax variants, ReLA allows a head to assign a score of zero to every item in the context, effectively "switching off" when no relevant information is found. The authors observed that this "null rate" correlates with data quality, being higher for low-quality or hallucinated sentence pairs. Furthermore, ReLA's cross-attention layers demonstrate higher sparsity and align more closely with human word-to-word alignments than standard models.
Conclusion
Rectified Linear Attention provides a simple, efficient path toward sparse and interpretable Transformers. By leveraging ReLU and standard normalization techniques, it achieves the benefits of sparsity—such as better alignment and the ability to ignore irrelevant context—without the computational costs of sorting-based methods. Future research will explore scaling ReLA to extremely long sequences and multi-source architectures where selective attention is paramount.
Summary updated