XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, Quoc V. Le

TL;DR

XLNet is a generalized autoregressive pretraining method that captures bidirectional context without the downsides of BERT’s masking approach. By maximizing the expected likelihood over all permutations of a sequence's factorization order, it achieves state-of-the-art results across 20 natural language processing tasks, including SQuAD, GLUE, and RACE.

What Problem Does This Paper Address?

Before XLNet, the NLP world was divided into two pretraining camps: Autoregressive (AR) models like GPT and Autoencoding (AE) models like BERT.

  1. AR models are great at estimating joint probabilities (the "flow" of language) but are typically uni-directional. They only see the "past" to predict the "future," making them less effective at tasks that require deep bidirectional context (like question answering).
  2. AE models (BERT) achieve bidirectionality by using [MASK] tokens. However, this creates two problems:
  3. Pretrain-Finetune Discrepancy: The model sees [MASK] tokens during training but never during actual use (finetuning), which can confuse the model.
  4. Independence Assumption: BERT assumes that masked tokens are independent of each other. For example, in the phrase "New York," if both words are masked, BERT tries to predict "New" and "York" separately, failing to model the strong dependency between them.

XLNet aims to combine the best of both worlds: the bidirectional context of BERT and the mathematical rigor of AR modeling.

Methods

The authors introduce two primary innovations to bridge the AR-AE gap:

  • Permutation Language Modeling (PLM): Instead of using a fixed left-to-right order, XLNet trains on all possible permutations of a sequence's factorization order. For a sequence of four tokens, there are $4!$ (24) possible orders. By training on these permutations, a specific token can eventually "see" tokens from both its left and right in the original sequence while still maintaining a valid autoregressive structure. Crucially, the actual sequence order remains the same; only the order in which the model "assembles" the probability changes.
  • Two-Stream Self-Attention: Standard Transformers can't handle PLM because if you want to predict a token at position $i$, the model needs to know the position $i$ but must not see the content of the token at $i$. XLNet solves this with two sets of hidden states:
Two-Stream Self-Attention Mechanism
  1. Content Stream: Standard attention that encodes both the position and the token content.
  2. Query Stream: Only has access to the target position and the context (previous tokens in the permutation), but not the content of the target token itself.
  3. Transformer-XL Integration: XLNet adopts the "segment recurrence" and "relative positional encoding" mechanisms from Transformer-XL, allowing the model to handle much longer dependencies than the original BERT.

Datasets

The authors used a massive corpus to scale up XLNet:

  • Standard: BooksCorpus and English Wikipedia (the same 13GB used for BERT).
  • Extended: Giga5 (16GB), ClueWeb 2012-B, and Common Crawl.
  • Total Scale: After filtering, the training data totaled 32.89 billion subword pieces, significantly larger than the original BERT training set.

Evaluation was performed on a wide array of benchmarks, including GLUE (General Language Understanding Evaluation), SQuAD (Stanford Question Answering Dataset), RACE (Reading Comprehension from Examinations), and various text classification tasks (IMDB, Yelp).

Key Insights and Ideas

  • Bidirectionality without Masking: The "aha moment" is that you don't need [MASK] tokens to see both sides of a word. By simply changing the order in which you calculate probabilities (permutations), the model naturally learns from the entire context.
  • Modeling Dependencies: Because XLNet is autoregressive, it uses the product rule of probability. This means it can model the relationship between multiple predicted tokens (like "New" and "York") rather than treating them as independent events.
  • Architecture Matters: The ablation studies show that while the permutation objective is the star, the Transformer-XL backbone (specifically the recurrence mechanism) is vital for performance on long-form text tasks like RACE.

Key Results

XLNet-Large consistently outperformed BERT-Large, often by a significant margin:

Performance Comparison: BERT vs. XLNet
  • SQuAD 2.0: XLNet achieved an F1 score of 90.6, compared to BERT’s 81.8.
  • RACE: On this difficult reading comprehension task, XLNet reached 85.4% accuracy, a massive jump from BERT’s 72.0%.
  • GLUE: XLNet set new records on almost all sub-tasks (MNLI, QNLI, etc.).
  • Fair Comparison: Even when trained on the exact same data as BERT, XLNet still outperformed it, proving that the improvement comes from the algorithm, not just the larger dataset.

Limitations and Future Work

  • Optimization Difficulty: Permutation language modeling is a much harder task for a neural network to learn than simple masking. To make it converge, the authors had to use "Partial Prediction," only predicting the last $1/K$ tokens in a permutation.
  • Computational Cost: Training XLNet-Large required 512 TPU v3 chips for 5.5 days. This level of compute is out of reach for most independent researchers.
  • Inference Speed: While the paper focuses on accuracy, the two-stream attention mechanism adds complexity to the architecture, which can impact training efficiency compared to simpler models.

Summary updated

Abstract

With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment setting, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.
BibTeX
@inproceedings{yang2019xlnet,
  author = {Zhilin Yang and Zihang Dai and Yiming Yang and Jaime G. Carbonell and Ruslan Salakhutdinov and Quoc V. Le},
  title = {{XLNet: Generalized Autoregressive Pretraining for Language Understanding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=TcRGCL0B4DG},
}
Figure