Abstract
With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves better performance than pretraining approaches based on autoregressive language modeling. However, relying on corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.
Introduction: The Best of Both Worlds
Before XLNet, the NLP landscape was primarily divided between two pretraining philosophies: Autoregressive (AR) modeling and Autoencoding (AE) modeling. AR models like GPT predict tokens sequentially, making them excellent at probability estimation but "blind" to future context. Conversely, AE models like BERT use Masked Language Modeling (MLM) to capture bidirectional context.
However, BERT's reliance on the [MASK] symbol introduces a pretrain-finetune discrepancy, as this symbol never appears in real-world data. Furthermore, BERT assumes masked tokens are independent of each other, failing to model dependencies in phrases like "New York." XLNet was designed to bridge this gap, achieving bidirectional understanding through an autoregressive framework.
Methodology: Permutation and Two-Stream Attention
XLNet's primary innovation is Permutation Language Modeling (PLM). Instead of masking tokens, XLNet shuffles the factorization order of the sequence while keeping the physical input order intact. This allows the model to incorporate context from both "past" and "future" positions in a mathematically rigorous autoregressive manner.
To implement PLM, the authors introduced Two-Stream Self-Attention. Standard Transformers cannot distinguish between predicting different tokens at the same position if they only receive positional information. XLNet solves this by maintaining two sets of hidden states:
- Content Stream ($h$): Encodes both the position and the actual token content.
- Query Stream ($g$): Encodes the target position but excludes the token content to prevent the model from "cheating" during prediction.
Additionally, XLNet integrates the "segment recurrence" and "relative positional encoding" mechanisms from Transformer-XL, enabling the model to handle long-range dependencies across document segments far more effectively than standard BERT architectures.
Datasets and Pretraining
XLNet was pretrained on a massive scale, utilizing 32.89 billion subword pieces. The training data included the standard BooksCorpus and English Wikipedia, augmented by Giga5, ClueWeb 2012-B, and Common Crawl. This diverse dataset allowed the model to learn robust representations across various domains.
Key Results and Performance
XLNet consistently outperformed BERT-Large and the subsequent RoBERTa model across a wide array of benchmarks. On the GLUE benchmark, XLNet set new state-of-the-art records for single-task models and multi-task ensembles.
Notable achievements include:
- SQuAD 2.0: XLNet reached an F1 score of 90.6, significantly higher than BERT's 81.8.
- RACE: On this difficult reading comprehension task, XLNet achieved 85.4% accuracy, a substantial improvement over BERT's 72.0%.
- Efficiency: Ablation studies demonstrated that even when trained on the same data as BERT, XLNet's objective provided a superior performance boost.
Conclusion
XLNet demonstrates that the limitations of masked language modeling can be overcome by a generalized autoregressive approach. By combining the strengths of AR and AE models through permutation-based training and two-stream attention, XLNet provides a more flexible and powerful framework for language understanding. While computationally intensive, the model's ability to capture bidirectional dependencies without data corruption marks a significant milestone in NLP research.
Summary updated