Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, Ruslan Salakhutdinov

Abstract

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning).

Introduction

Standard Transformer architectures process text in fixed-length segments (e.g., 512 tokens). This approach introduces two significant limitations: limited dependency length and context fragmentation. Because the model cannot attend to tokens outside its current segment, it fails to capture long-range relationships. Furthermore, segments are often created by simple truncation without regard for semantic boundaries, leaving the model without sufficient context to predict the initial tokens of a new segment. While Recurrent Neural Networks (RNNs) can theoretically handle long sequences, they suffer from vanishing gradients and lack the parallelization benefits of Transformers.

Methodology

Transformer-XL introduces two primary innovations to overcome the fixed-length barrier: Segment-Level Recurrence and Relative Positional Encodings.

Segment-Level Recurrence

To enable information flow across segments, Transformer-XL "caches" the hidden states of the previous segment. During the processing of the current segment, the attention mechanism can look back at these cached states. This creates a recurrent connection at the segment level. Unlike RNNs, where recurrence is applied to individual tokens, this mechanism allows the model to capture dependencies across thousands of tokens while maintaining the computational efficiency of self-attention.

Segment-level recurrence mechanism in training and evaluation phases.

Relative Positional Encodings

Reusing hidden states from previous segments introduces a challenge for standard absolute positional encodings. If "Position 1" always refers to the start of a segment, the model cannot distinguish between "Position 1" in the current segment and "Position 1" in the cached memory. Transformer-XL resolves this by using relative positional encodings. Instead of absolute indices, the model learns to attend based on the relative distance between tokens (e.g., "10 steps prior"). This ensures that the temporal coherence of the "memory" is preserved across segment boundaries.

Experiments and Results

The model was evaluated on several benchmarks, including WikiText-103, enwiki8, text8, One Billion Word, and Penn Treebank.

  • State-of-the-Art Performance: Transformer-XL achieved a perplexity of 18.3 on WikiText-103 and became the first model to reach a bits-per-character (bpc) of 0.99 on enwiki8.
  • Extended Context: The architecture captures dependencies roughly 450% longer than vanilla Transformers and 80% longer than LSTMs.
  • Evaluation Efficiency: By reusing representations from previous segments, Transformer-XL avoids redundant computations during inference, resulting in an evaluation speedup of over 1,800x compared to vanilla Transformers.
  • Text Generation: When trained on WikiText-103, the model generates highly coherent, novel articles spanning thousands of tokens, demonstrating its ability to maintain long-term structure.

Conclusion

Transformer-XL successfully bridges the gap between the parallelizable power of Transformers and the long-term persistence of recurrent models. By implementing segment-level recurrence and relative positional encodings, it effectively solves the context fragmentation problem and significantly extends the effective context window. While limited by GPU memory for caching, the architecture sets a new standard for long-form language modeling and opens doors for applications in document summarization and long-sequence processing.

Summary updated

Abstract

Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably coherent, novel text articles with thousands of tokens. Our code, pretrained models, and hyperparameters are available in both Tensorflow and PyTorch.
BibTeX
@inproceedings{dai2019transformer,
  author = {Zihang Dai and Zhilin Yang and Yiming Yang and Jaime G. Carbonell and Quoc Viet Le and Ruslan Salakhutdinov},
  title = {{Transformer-XL: Attentive Language Models beyond a Fixed-Length Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=enTGbgh32D5},
}
Figure