On the Role of Bidirectionality in Language Model Pre-Training

Mikel Artetxe, Jingfei Du, Naman Goyal, Luke Zettlemoyer, Ves Stoyanov

Abstract

Prior work on language model pre-training has explored diverse architectures and learning objectives, but differences in data, hyperparameters, and evaluation protocols make principled comparisons difficult. This study isolates bidirectionality as a key factor differentiating existing approaches. We present a comprehensive study of its role in next-token prediction, text infilling, zero-shot priming, and fine-tuning. By introducing a framework that generalizes models like GPT, BERT, and Prefix LMs, we distinguish between bidirectional context and bidirectional attention. Our findings reveal that the optimal configuration is application-dependent: bidirectional attention benefits fine-tuning and infilling but degrades next-token prediction and zero-shot priming. These trade-offs remain consistent as models scale to 6.7 billion parameters.

Introduction

In the landscape of Large Language Models (LLMs), two dominant paradigms have emerged: unidirectional models (e.g., GPT), which process text strictly from left to right, and bidirectional models (e.g., BERT), which observe the full context simultaneously. While both have achieved remarkable success, comparing them fairly is challenging. Research groups often use different datasets, compute budgets, and hyperparameters, leaving it unclear whether GPT’s generative prowess stems from its architecture or its massive training scale.

This research addresses this gap by isolating "bidirectionality" as the primary independent variable. By controlling for data and compute, we investigate how different attention and context mechanisms impact performance across a spectrum of NLP tasks.

A Unified Framework for Bidirectionality

To facilitate an apples-to-apples comparison, the authors propose a generalized pre-training framework using three controllable parameters:

  1. $n_{bidir}$ (Bidirectional Attention): Determines how many initial tokens in a sequence can attend to each other non-causally (similar to a Prefix LM).
  2. $n_{mask}$ (Bidirectional Context): Determines how many tokens are masked and moved to the end of the sequence to be predicted (infilling).
  3. $n_{predict}$ (Supervision): Determines the number of tokens the model is tasked with predicting during training.

By adjusting these "knobs," the study defines six model variants, including NXTUNI (standard unidirectional GPT-style), MSKBI (standard bidirectional BERT-style), and HYBPRE (a hybrid model utilizing a bidirectional prefix). All models were trained on the same 112-billion-token English corpus, spanning five scales up to 6.7 billion parameters.

Experimental Results

1. The Trade-off in Attention

The study distinguishes between bidirectional context (access to future tokens) and bidirectional attention (the ability for tokens to attend to each other in a non-causal way). The results indicate that bidirectional attention is a double-edged sword. While it is the primary driver for high performance in Natural Language Understanding (NLU) tasks via fine-tuning, it acts as a penalty for generative tasks. Even when a model is provided with a bidirectional prefix to assist in prompt understanding, it consistently performs worse at next-token prediction than a purely unidirectional model.

2. Task-Specific Performance

  • Language Modeling & Zero-Shot Priming: The unidirectional NXTUNI model was the clear leader. Introducing any form of bidirectionality during training increased perplexity, suggesting that causal constraints are beneficial for learning the underlying distribution of language for generation.
  • Fine-tuning: The bidirectional MSKBI model dominated the GLUE benchmark. At the 6.7B scale, the performance gap between bidirectional and unidirectional models remained significant, favoring the BERT-style approach for discriminative tasks.
  • Infilling: While models trained specifically with masking (MSKBI) were superior at direct infilling, unidirectional models proved surprisingly competitive when used to score multiple candidates in a zero-shot setting.

3. Consistency at Scale

A critical finding of this work is that scaling does not bridge the architectural gap. The performance differences between unidirectional and bidirectional models remained consistent from 125 million to 6.7 billion parameters. This suggests that the trade-offs inherent in these architectures are fundamental and cannot be overcome simply by increasing model size or training data.

Discussion and Limitations

While comprehensive, the study notes several constraints. The masking strategy was limited to individual tokens rather than spans (as seen in T5 or SpanBERT), and the research focused on decoder-only or prefix-decoder architectures, excluding encoder-decoder setups. Additionally, the masking rate was fixed at 15%, leaving the impact of masking density on scaling as a subject for future inquiry.

Conclusion

The research demonstrates that there is no "silver bullet" architecture for language modeling. The choice between unidirectional and bidirectional structures involves a fundamental trade-off between generative fluency and discriminative power. As the field moves toward larger models, these results suggest that modular architectures—capable of switching attention modes based on the task—may be the most promising path forward for developing versatile AI systems.

Summary updated

Abstract

Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. In this work, we focus on bidirectionality as a key factor that differentiates existing approaches, and present a comprehensive study of its role in next token prediction, text infilling, zero-shot priming and fine-tuning. We propose a new framework that generalizes prior approaches, including fully unidirectional models like GPT, fully bidirectional models like BERT, and hybrid models like CM3 and prefix LM. Our framework distinguishes between two notions of bidirectionality (bidirectional context and bidirectional attention) and allows us to control each of them separately. We find that the optimal configuration is largely application-dependent (e.g., bidirectional attention is beneficial for fine-tuning and infilling, but harmful for next token prediction and zero-shot priming). We train models with up to 6.7B parameters, and find differences to remain consistent at scale. While prior work on scaling has focused on left-to-right autoregressive models, our results suggest that this approach comes with some trade-offs, and it might be worthwhile to develop very large bidirectional models.
BibTeX
@inproceedings{artetxe2022role,
  author = {Mikel Artetxe and Jingfei Du and Naman Goyal and Luke Zettlemoyer and Ves Stoyanov},
  title = {{On the Role of Bidirectionality in Language Model Pre-Training}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=w_rCUBW4RX5},
}
Figure