Sparse Attention with Linear Units

Biao Zhang, Ivan Titov, Rico Sennrich

Abstract

Standard Transformer models rely on the softmax function to compute attention weights, which forces a dense probability distribution where every token receives some non-zero weight. This paper introduces Rectified Linear Attention (ReLA), a sparse alternative that replaces softmax with a simple ReLU activation. ReLA achieves natural sparsity, allowing attention heads to "switch off" when context is irrelevant. Experimental results on multiple WMT translation benchmarks show that ReLA matches the performance of vanilla Transformers while offering superior decoding efficiency and interpretability compared to existing sparse attention methods.

Introduction

In modern Transformer architectures, the softmax function is the standard mechanism for normalizing attention scores. However, softmax has two significant drawbacks: it is inherently dense, assigning weight to even irrelevant noise, and it is computationally rigid, requiring every query to attend to the context with a total mass of exactly 1.0. While previous sparse attention methods like sparsemax or entmax address density, they often introduce complex sorting operations that slow down training and inference. This work explores whether sparsity can be achieved more efficiently using simple linear units.

Rectified Linear Attention (ReLA)

The core proposal is ReLA (Rectified Linear Attention), which replaces the standard softmax pipeline with a ReLU (Rectified Linear Unit) activation:

$$\alpha = \text{ReLU}(f(Q, K^T))$$

Because ReLU outputs zero for any negative input, sparsity emerges naturally without the need for thresholding or sorting. To stabilize training and prevent gradient explosions caused by the removal of softmax normalization, the authors propose two variants:

  • ReLA-i: Employs RMSNorm on the attention output combined with a specialized Xavier initialization.
  • ReLA-g: Introduces a gating function (a sigmoid-weighted unit) during normalization to provide flexible output scaling.
Comparison between standard Softmax attention and the proposed ReLA-g architecture.

Unlike softmax, ReLA abandons the probabilistic constraint. A query can attend to context with a total weight greater than one or, uniquely, a total weight of zero—effectively "switching off" the attention head for that token.

Experimental Evaluation

The model was evaluated on five major Machine Translation (MT) benchmarks, including WMT14 (En-De, En-Fr), WMT18 (En-Fi, Zh-En), and WMT16 (Ro-En). These datasets range from 608K to 36M sentence pairs, testing the architecture's scalability across different data regimes.

Key Results and Insights

Performance and Efficiency

ReLA achieves BLEU scores comparable to vanilla Transformers. For instance, on WMT14 En-De, ReLA-g scored 26.6 compared to the baseline's 26.9, while outperforming other sparse variants. Crucially, ReLA maintains $O(n)$ complexity, making it significantly faster than sorting-based sparse methods.

Decoding speedup of ReLA compared to other sparse attention mechanisms across various sequence lengths.

As shown in the decoding speed analysis, ReLA is at least 1.8x faster than sparsemax or entmax, with speeds nearly identical to the vanilla softmax Transformer across various sequence lengths.

Interpretability and Data Sensing

ReLA's sparse weights are more linguistically meaningful than softmax. It achieved an Alignment Error Rate (AER) of 64.46, significantly lower than softmax's 75.95, indicating better alignment with human-annotated word mappings. Furthermore, the authors observed that "null attention" rates (where heads switch off) spike when the model encounters noisy or hallucinated data, suggesting ReLA can serve as an internal gauge for data quality.

Conclusion

Rectified Linear Attention provides a simple, efficient, and interpretable alternative to softmax. By leveraging the natural sparsity of ReLU, it avoids the computational overhead of previous sparse attention techniques while enabling new capabilities like null attention. Future work will explore ReLA's application to extremely long sequences and multi-source architectures where selective ignoring of input sources is critical.

Summary updated

Abstract

Recently, it has been argued that encoder-decoder models can be made more interpretable by replacing the softmax function in the attention with its sparse variants. In this work, we introduce a novel, simple method for achieving sparsity in attention: we replace the softmax activation with a ReLU, and show that sparsity naturally emerges from such a formulation. Training stability is achieved with layer normalization with either a specialized initialization or an additional gating function. Our model, which we call Rectified Linear Attention (ReLA), is easy to implement and more efficient than previously proposed sparse attention mechanisms. We apply ReLA to the Transformer and conduct experiments on five machine translation tasks. ReLA achieves translation performance comparable to several strong baselines, with training and decoding speed similar to that of the vanilla attention. Our analysis shows that ReLA delivers high sparsity rate and head diversity, and the induced cross attention achieves better accuracy with respect to source-target word alignment than recent sparsified softmax-based models. Intriguingly, ReLA heads also learn to attend to nothing (i.e. 'switch off') for some queries, which is not possible with sparsified softmax alternatives.
BibTeX
@inproceedings{zhang2021sparse,
  author = {Biao Zhang and Ivan Titov and Rico Sennrich},
  title = {{Sparse Attention with Linear Units}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=tdO1VcgBw3},
}
Figure