Abstract
Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning).
The Challenge of Fixed-Length Context
Standard "vanilla" Transformers process text in isolated, fixed-length segments. This approach introduces two significant limitations: first, the model cannot capture dependencies longer than the predefined segment length (e.g., 512 tokens). Second, it suffers from context fragmentation. Because segments are typically created by chunking text without regard for semantic boundaries, the model lacks the necessary context to predict the initial tokens of a new segment, leading to inefficient optimization.
During evaluation, vanilla Transformers often use a sliding window approach to mitigate fragmentation, but this requires recomputing the entire hidden state for every single new token, making it computationally prohibitive for long-range tasks.
Methodology: Segment-Level Recurrence
To overcome these limitations, Transformer-XL introduces a segment-level recurrence mechanism. Instead of discarding hidden states from previous segments, the model caches them. When processing a new segment, these cached states are treated as an extended, "read-only" memory that the current segment can attend to.
Crucially, gradients are not backpropagated into the cached memory (a "stop-gradient" operation), which maintains computational efficiency while allowing the effective context to grow linearly with the number of layers ($O(Layer \times Segment\_Length)$). This recurrence ensures that the model can "warm up" using information from the past, effectively solving the context fragmentation problem.
Relative Positional Encoding
Reusing hidden states across segments introduces a challenge for standard absolute positional encodings. If the 1st token of the current segment and the 1st token of the cached segment both use the same absolute position index, the model cannot distinguish their temporal order. Transformer-XL resolves this with a novel relative positional encoding scheme. By encoding the relative distance between tokens rather than their absolute positions, the model maintains temporal coherence across the extended context and generalizes to longer sequences during evaluation than those seen during training.
Key Results and Performance
Transformer-XL achieved new state-of-the-art (SOTA) results across several major benchmarks:
- WikiText-103: Reduced perplexity from 20.5 to 18.3.
- enwiki8: Improved bits-per-character (bpc) to 0.99.
- Efficiency: Due to state reuse, Transformer-XL is up to 1,800x faster than vanilla Transformers during evaluation.
Even on datasets like the "One Billion Word" benchmark, where sentences are shuffled and long-term context is unavailable, Transformer-XL outperformed baselines. This demonstrates that the architecture's benefits extend beyond long-range memory, successfully addressing the performance dip caused by context fragmentation at segment boundaries.
Conclusion
Transformer-XL represents a significant leap in language modeling by enabling Transformers to capture dependencies significantly longer than both RNNs and vanilla Transformers. By combining segment-level recurrence with relative positional encoding, the architecture provides a robust solution for long-range sequence modeling that is both more accurate and vastly more efficient during inference.
Summary updated