On the Role of Bidirectionality in Language Model Pre-Training

Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, Veselin Stoyanov

Introduction

The landscape of large language model (LLM) pre-training is diverse, featuring causal models like GPT, masked models like BERT, and hybrid architectures like T5. While these models vary significantly, researchers at Meta AI argue that the core differentiator is "bidirectionality." However, bidirectionality is often treated as a monolithic feature. This study deconstructs it into two distinct mechanisms—bidirectional context and bidirectional attention—to evaluate their individual impacts on performance across varying model scales.

A Unified Framework for Bidirectionality

To isolate the effects of different architectural choices, the authors propose a unified framework that generalizes existing pre-training approaches using three controllable parameters, or "knobs":

  1. $n_{bidir}$ (Bidirectional Attention): Defines a prefix length where tokens can attend to each other freely, similar to a BERT encoder or a Prefix LM.
  2. $n_{mask}$ (Bidirectional Context): Determines the number of tokens masked and moved to the end of the sequence, allowing them to be predicted using both left and right context.
  3. $n_{predict}$ (Supervision): Specifies the number of tokens the model is tasked with predicting during the pre-training phase.
The unified framework deconstructs bidirectionality into three parameters: bidirectional attention, bidirectional context, and supervision.

By adjusting these parameters, the researchers created six model variants, ranging from standard unidirectional models (NXTUNI) to fully bidirectional generative decoders (MSKBI). These variants were trained across five sizes (125M to 6.7B parameters) on a consistent 100B token budget.

Experimental Setup

The models were pre-trained on a massive 112B token corpus (including BookCorpus, CC-News, and OpenWebText). Evaluation spanned four key areas:

  • Language Modeling: Measured via suffix perplexity on held-out data.
  • Zero-shot Priming: Multiple-choice tasks (e.g., HellaSwag, PIQA).
  • Fine-tuning: Performance on the GLUE benchmark.
  • Infilling: Accuracy in predicting single masked words.

The study reveals a fundamental "no free lunch" trade-off in language model architecture. Bidirectional mechanisms significantly enhance performance in some areas while degrading it in others.

Performance comparison across model sizes for different variants. Bidirectional models (MskBi) excel at fine-tuning and infilling, while unidirectional models (NxtUni) dominate zero-shot accuracy and language modeling.

Fine-tuning vs. Zero-shot Priming

Bidirectional attention is a major advantage for fine-tuning tasks. At the 6.7B scale, the fully bidirectional MSKBI achieved an average GLUE score of 91.0, outperforming the unidirectional NXTUNI (88.6). Conversely, bidirectional attention in the prefix (Prefix LM) led to a noticeable decline in zero-shot accuracy. NXTUNI consistently remained the superior choice for zero-shot priming and standard next-token generation.

The Persistence of Architectural Gaps

A critical finding is that scaling does not bridge these performance gaps. As models grew from 125M to 6.7B parameters, the relative strengths and weaknesses of each architecture remained remarkably consistent. This suggests that architectural choices regarding bidirectionality impose fundamental constraints that cannot be overcome simply by increasing parameter counts.

Language Modeling and Infilling

Unidirectional models achieved the best (lowest) perplexity. The authors attribute the lag in bidirectional models to a lower "density" of training signal, as they often predict fewer tokens per sequence. In infilling tasks, while BERT-style models naturally lead, standard GPT-style models proved surprisingly competitive when used to score candidates rather than generate them.

Discussion and Limitations

The research highlights that the current industry focus on purely causal models (like GPT-4) may overlook potential gains in Natural Language Understanding (NLU) tasks where bidirectional models excel. However, bidirectional pre-training currently faces challenges in sample efficiency due to predicting fewer tokens per sequence. Future work could explore more efficient masking strategies and traditional encoder-decoder architectures to further refine these insights.

Ultimately, the choice of bidirectionality should be driven by the intended application: bidirectional models for NLU and fine-tuning, and unidirectional models for generation and zero-shot tasks.

Summary updated

Abstract

Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. In this work, we focus on bidirectionality as a key factor that differentiates existing approaches, and present a comprehensive study of its role in next token prediction, text infilling, zero-shot priming and fine-tuning. We propose a new framework that generalizes prior approaches, including fully unidirectional models like GPT, fully bidirectional models like BERT, and hybrid models like CM3 and prefix LM. Our framework distinguishes between two notions of bidirectionality (bidirectional context and bidirectional attention) and allows us to control each of them separately. We find that the optimal configuration is largely application-dependent (e.g., bidirectional attention is beneficial for fine-tuning and infilling, but harmful for next token prediction and zero-shot priming). We train models with up to 6.7B parameters, and find differences to remain consistent at scale. While prior work on scaling has focused on left-to-right autoregressive models, our results suggest that this approach comes with some trade-offs, and it might be worthwhile to develop very large bidirectional models.
BibTeX
@inproceedings{artetxe2022role,
  author = {Mikel Artetxe and Jingfei Du and Naman Goyal and Luke Zettlemoyer and Veselin Stoyanov},
  title = {{On the Role of Bidirectionality in Language Model Pre-Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=w_rCUBW4RX5},
}
Figure