Abstract
Prior work on language model pre-training has explored diverse architectures and learning objectives, but differences in data, hyperparameters, and evaluation protocols make principled comparisons difficult. This study isolates bidirectionality as a key factor differentiating existing approaches. We present a comprehensive study of its role in next-token prediction, text infilling, zero-shot priming, and fine-tuning. By introducing a framework that generalizes models like GPT, BERT, and Prefix LMs, we distinguish between bidirectional context and bidirectional attention. Our findings reveal that the optimal configuration is application-dependent: bidirectional attention benefits fine-tuning and infilling but degrades next-token prediction and zero-shot priming. These trade-offs remain consistent as models scale to 6.7 billion parameters.
Introduction
In the landscape of Large Language Models (LLMs), two dominant paradigms have emerged: unidirectional models (e.g., GPT), which process text strictly from left to right, and bidirectional models (e.g., BERT), which observe the full context simultaneously. While both have achieved remarkable success, comparing them fairly is challenging. Research groups often use different datasets, compute budgets, and hyperparameters, leaving it unclear whether GPT’s generative prowess stems from its architecture or its massive training scale.
This research addresses this gap by isolating "bidirectionality" as the primary independent variable. By controlling for data and compute, we investigate how different attention and context mechanisms impact performance across a spectrum of NLP tasks.
A Unified Framework for Bidirectionality
To facilitate an apples-to-apples comparison, the authors propose a generalized pre-training framework using three controllable parameters:
- $n_{bidir}$ (Bidirectional Attention): Determines how many initial tokens in a sequence can attend to each other non-causally (similar to a Prefix LM).
- $n_{mask}$ (Bidirectional Context): Determines how many tokens are masked and moved to the end of the sequence to be predicted (infilling).
- $n_{predict}$ (Supervision): Determines the number of tokens the model is tasked with predicting during training.
By adjusting these "knobs," the study defines six model variants, including NXTUNI (standard unidirectional GPT-style), MSKBI (standard bidirectional BERT-style), and HYBPRE (a hybrid model utilizing a bidirectional prefix). All models were trained on the same 112-billion-token English corpus, spanning five scales up to 6.7 billion parameters.
Experimental Results
1. The Trade-off in Attention
The study distinguishes between bidirectional context (access to future tokens) and bidirectional attention (the ability for tokens to attend to each other in a non-causal way). The results indicate that bidirectional attention is a double-edged sword. While it is the primary driver for high performance in Natural Language Understanding (NLU) tasks via fine-tuning, it acts as a penalty for generative tasks. Even when a model is provided with a bidirectional prefix to assist in prompt understanding, it consistently performs worse at next-token prediction than a purely unidirectional model.
2. Task-Specific Performance
- Language Modeling & Zero-Shot Priming: The unidirectional NXTUNI model was the clear leader. Introducing any form of bidirectionality during training increased perplexity, suggesting that causal constraints are beneficial for learning the underlying distribution of language for generation.
- Fine-tuning: The bidirectional MSKBI model dominated the GLUE benchmark. At the 6.7B scale, the performance gap between bidirectional and unidirectional models remained significant, favoring the BERT-style approach for discriminative tasks.
- Infilling: While models trained specifically with masking (MSKBI) were superior at direct infilling, unidirectional models proved surprisingly competitive when used to score multiple candidates in a zero-shot setting.
3. Consistency at Scale
A critical finding of this work is that scaling does not bridge the architectural gap. The performance differences between unidirectional and bidirectional models remained consistent from 125 million to 6.7 billion parameters. This suggests that the trade-offs inherent in these architectures are fundamental and cannot be overcome simply by increasing model size or training data.
Discussion and Limitations
While comprehensive, the study notes several constraints. The masking strategy was limited to individual tokens rather than spans (as seen in T5 or SpanBERT), and the research focused on decoder-only or prefix-decoder architectures, excluding encoder-decoder setups. Additionally, the masking rate was fixed at 15%, leaving the impact of masking density on scaling as a subject for future inquiry.
Conclusion
The research demonstrates that there is no "silver bullet" architecture for language modeling. The choice between unidirectional and bidirectional structures involves a fundamental trade-off between generative fluency and discriminative power. As the field moves toward larger models, these results suggest that modular architectures—capable of switching attention modes based on the task—may be the most promising path forward for developing versatile AI systems.
Summary updated