Abstract
Transformer is the backbone of modern NLP models. In this paper, we propose RealFormer, a simple and generic technique to create Residual Attention Layer Transformer networks that significantly outperform the canonical Transformer and its variants (BERT, ETC, etc.) on a wide spectrum of tasks including Masked Language Modeling, GLUE, SQuAD, Neural Machine Translation, WikiHop, HotpotQA, Natural Questions, and OpenKP. We also observe empirically that RealFormer stabilizes training and leads to models with sparser attention.
Introduction
In the world of Natural Language Processing, the Transformer architecture is the dominant paradigm. From BERT to GPT, state-of-the-art models rely on the ability to relate different parts of a sequence through multi-head attention. However, while residual (skip) connections have been perfected for token embeddings, the attention mechanism itself has largely been left to start from scratch at every layer. Standard Transformers typically utilize either Post-LN (Layer Norm after the sub-layer) or Pre-LN (Layer Norm before) designs. While Pre-LN is more stable for training massive models, it often performs slightly worse than Post-LN on downstream tasks.
RealFormer addresses this by introducing a "direct path" for attention scores, allowing the contextual relationships identified in one layer to persist and be refined in subsequent layers.
Methodology: Residual Attention
The core innovation is the Residual Attention Layer. In a standard Transformer, the attention score is calculated as: $$Attention(Q, K, V) = Softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
In RealFormer, the attention score for layer $L$ incorporates the raw, pre-softmax attention score from the previous layer ($Prev$): $$RealAttention(Q, K, V, Prev) = Softmax\left(\frac{QK^T}{\sqrt{d_k}} + Prev\right)V$$
This architectural tweak adds zero new parameters and can be applied to encoders, decoders, and cross-attention modules. For extremely deep networks, a "running mean" of scores can be used to prevent value explosion.
Experimental Results
The authors evaluated RealFormer across a massive battery of benchmarks, including GLUE, SQuAD, and WMT’14 translation tasks.
Pre-training Efficiency
A striking finding is the efficiency gain during pre-training. RealFormer consistently achieves higher Masked Language Modeling (MLM) accuracy across all model sizes (Small, Base, Large, and xLarge) compared to both Post-LN and Pre-LN baselines.
Key performance highlights include:
- Training Speed: A RealFormer model pre-trained for 500,000 steps can outperform a standard Post-LN Transformer pre-trained for 1,000,000 steps on GLUE.
- State-of-the-Art Translation: RealFormer achieved a new SOTA BLEU score of 43.97 on WMT’14 En-Fr (without extra data).
- Leaderboard Success: The RealFormer-enhanced ETC model ranked #1 on the WikiHop leaderboard at the time of release.
Discussion and Insights
- Sparsity and Confidence: Qualitatively, RealFormer leads to sparser attention in higher layers. The model becomes more selective about which tokens to attend to as it gets deeper.
- Inter-layer Correlation: Attention heads in RealFormer are more similar to their counterparts in adjacent layers than in standard Transformers, suggesting the residual connection helps the model refine a consistent view of the data.
- Stability at Scale: For "xLarge" models (~1B parameters), standard Post-LN Transformers often become unstable. RealFormer remains stable, acting as a regularizer that facilitates scaling.
Conclusion
RealFormer provides a simple yet powerful modification to the Transformer architecture. By allowing attention scores to flow through the network via residual connections, it improves performance, training stability, and efficiency across a wide range of NLP tasks without increasing parameter count. Future work may explore specialized initialization schemes and hardware-specific optimizations to further leverage this residual attention path.
Summary updated