Abstract
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning).
Introduction
Standard Transformer architectures process text in fixed-length segments (e.g., 512 tokens). This approach introduces two significant limitations: limited dependency length and context fragmentation. Because the model cannot attend to tokens outside its current segment, it fails to capture long-range relationships. Furthermore, segments are often created by simple truncation without regard for semantic boundaries, leaving the model without sufficient context to predict the initial tokens of a new segment. While Recurrent Neural Networks (RNNs) can theoretically handle long sequences, they suffer from vanishing gradients and lack the parallelization benefits of Transformers.
Methodology
Transformer-XL introduces two primary innovations to overcome the fixed-length barrier: Segment-Level Recurrence and Relative Positional Encodings.
Segment-Level Recurrence
To enable information flow across segments, Transformer-XL "caches" the hidden states of the previous segment. During the processing of the current segment, the attention mechanism can look back at these cached states. This creates a recurrent connection at the segment level. Unlike RNNs, where recurrence is applied to individual tokens, this mechanism allows the model to capture dependencies across thousands of tokens while maintaining the computational efficiency of self-attention.
Relative Positional Encodings
Reusing hidden states from previous segments introduces a challenge for standard absolute positional encodings. If "Position 1" always refers to the start of a segment, the model cannot distinguish between "Position 1" in the current segment and "Position 1" in the cached memory. Transformer-XL resolves this by using relative positional encodings. Instead of absolute indices, the model learns to attend based on the relative distance between tokens (e.g., "10 steps prior"). This ensures that the temporal coherence of the "memory" is preserved across segment boundaries.
Experiments and Results
The model was evaluated on several benchmarks, including WikiText-103, enwiki8, text8, One Billion Word, and Penn Treebank.
- State-of-the-Art Performance: Transformer-XL achieved a perplexity of 18.3 on WikiText-103 and became the first model to reach a bits-per-character (bpc) of 0.99 on enwiki8.
- Extended Context: The architecture captures dependencies roughly 450% longer than vanilla Transformers and 80% longer than LSTMs.
- Evaluation Efficiency: By reusing representations from previous segments, Transformer-XL avoids redundant computations during inference, resulting in an evaluation speedup of over 1,800x compared to vanilla Transformers.
- Text Generation: When trained on WikiText-103, the model generates highly coherent, novel articles spanning thousands of tokens, demonstrating its ability to maintain long-term structure.
Conclusion
Transformer-XL successfully bridges the gap between the parallelizable power of Transformers and the long-term persistence of recurrent models. By implementing segment-level recurrence and relative positional encodings, it effectively solves the context fragmentation problem and significantly extends the effective context window. While limited by GPU memory for caching, the architecture sets a new standard for long-form language modeling and opens doors for applications in document summarization and long-sequence processing.
Summary updated