Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc V. Le, Ruslan Salakhutdinov

Abstract

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning).

The Challenge of Fixed-Length Context

Standard "vanilla" Transformers process text in isolated, fixed-length segments. This approach introduces two significant limitations: first, the model cannot capture dependencies longer than the predefined segment length (e.g., 512 tokens). Second, it suffers from context fragmentation. Because segments are typically created by chunking text without regard for semantic boundaries, the model lacks the necessary context to predict the initial tokens of a new segment, leading to inefficient optimization.

Vanilla Transformer training and evaluation phases showing fixed-length segments and context fragmentation.

During evaluation, vanilla Transformers often use a sliding window approach to mitigate fragmentation, but this requires recomputing the entire hidden state for every single new token, making it computationally prohibitive for long-range tasks.

Methodology: Segment-Level Recurrence

To overcome these limitations, Transformer-XL introduces a segment-level recurrence mechanism. Instead of discarding hidden states from previous segments, the model caches them. When processing a new segment, these cached states are treated as an extended, "read-only" memory that the current segment can attend to.

Transformer-XL segment-level recurrence mechanism, where hidden states from previous segments are cached to provide extended context.

Crucially, gradients are not backpropagated into the cached memory (a "stop-gradient" operation), which maintains computational efficiency while allowing the effective context to grow linearly with the number of layers ($O(Layer \times Segment\_Length)$). This recurrence ensures that the model can "warm up" using information from the past, effectively solving the context fragmentation problem.

Relative Positional Encoding

Reusing hidden states across segments introduces a challenge for standard absolute positional encodings. If the 1st token of the current segment and the 1st token of the cached segment both use the same absolute position index, the model cannot distinguish their temporal order. Transformer-XL resolves this with a novel relative positional encoding scheme. By encoding the relative distance between tokens rather than their absolute positions, the model maintains temporal coherence across the extended context and generalizes to longer sequences during evaluation than those seen during training.

Key Results and Performance

Transformer-XL achieved new state-of-the-art (SOTA) results across several major benchmarks:

  • WikiText-103: Reduced perplexity from 20.5 to 18.3.
  • enwiki8: Improved bits-per-character (bpc) to 0.99.
  • Efficiency: Due to state reuse, Transformer-XL is up to 1,800x faster than vanilla Transformers during evaluation.

Even on datasets like the "One Billion Word" benchmark, where sentences are shuffled and long-term context is unavailable, Transformer-XL outperformed baselines. This demonstrates that the architecture's benefits extend beyond long-range memory, successfully addressing the performance dip caused by context fragmentation at segment boundaries.

Conclusion

Transformer-XL represents a significant leap in language modeling by enabling Transformers to capture dependencies significantly longer than both RNNs and vanilla Transformers. By combining segment-level recurrence with relative positional encoding, the architecture provides a robust solution for long-range sequence modeling that is both more accurate and vastly more efficient during inference.

Summary updated

Abstract

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.
BibTeX
@inproceedings{dai2019transformer,
  author = {Zihang Dai and Zhilin Yang and Yiming Yang and Jaime G. Carbonell and Quoc V. Le and Ruslan Salakhutdinov},
  title = {{Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=enTGbgh32D5},
}
Figure