Introduction
The landscape of large language model (LLM) pre-training is diverse, featuring causal models like GPT, masked models like BERT, and hybrid architectures like T5. While these models vary significantly, researchers at Meta AI argue that the core differentiator is "bidirectionality." However, bidirectionality is often treated as a monolithic feature. This study deconstructs it into two distinct mechanisms—bidirectional context and bidirectional attention—to evaluate their individual impacts on performance across varying model scales.
A Unified Framework for Bidirectionality
To isolate the effects of different architectural choices, the authors propose a unified framework that generalizes existing pre-training approaches using three controllable parameters, or "knobs":
- $n_{bidir}$ (Bidirectional Attention): Defines a prefix length where tokens can attend to each other freely, similar to a BERT encoder or a Prefix LM.
- $n_{mask}$ (Bidirectional Context): Determines the number of tokens masked and moved to the end of the sequence, allowing them to be predicted using both left and right context.
- $n_{predict}$ (Supervision): Specifies the number of tokens the model is tasked with predicting during the pre-training phase.
By adjusting these parameters, the researchers created six model variants, ranging from standard unidirectional models (NXTUNI) to fully bidirectional generative decoders (MSKBI). These variants were trained across five sizes (125M to 6.7B parameters) on a consistent 100B token budget.
Experimental Setup
The models were pre-trained on a massive 112B token corpus (including BookCorpus, CC-News, and OpenWebText). Evaluation spanned four key areas:
- Language Modeling: Measured via suffix perplexity on held-out data.
- Zero-shot Priming: Multiple-choice tasks (e.g., HellaSwag, PIQA).
- Fine-tuning: Performance on the GLUE benchmark.
- Infilling: Accuracy in predicting single masked words.
Key Results and Scaling Trends
The study reveals a fundamental "no free lunch" trade-off in language model architecture. Bidirectional mechanisms significantly enhance performance in some areas while degrading it in others.
Fine-tuning vs. Zero-shot Priming
Bidirectional attention is a major advantage for fine-tuning tasks. At the 6.7B scale, the fully bidirectional MSKBI achieved an average GLUE score of 91.0, outperforming the unidirectional NXTUNI (88.6). Conversely, bidirectional attention in the prefix (Prefix LM) led to a noticeable decline in zero-shot accuracy. NXTUNI consistently remained the superior choice for zero-shot priming and standard next-token generation.
The Persistence of Architectural Gaps
A critical finding is that scaling does not bridge these performance gaps. As models grew from 125M to 6.7B parameters, the relative strengths and weaknesses of each architecture remained remarkably consistent. This suggests that architectural choices regarding bidirectionality impose fundamental constraints that cannot be overcome simply by increasing parameter counts.
Language Modeling and Infilling
Unidirectional models achieved the best (lowest) perplexity. The authors attribute the lag in bidirectional models to a lower "density" of training signal, as they often predict fewer tokens per sequence. In infilling tasks, while BERT-style models naturally lead, standard GPT-style models proved surprisingly competitive when used to score candidates rather than generate them.
Discussion and Limitations
The research highlights that the current industry focus on purely causal models (like GPT-4) may overlook potential gains in Natural Language Understanding (NLU) tasks where bidirectional models excel. However, bidirectional pre-training currently faces challenges in sample efficiency due to predicting fewer tokens per sequence. Future work could explore more efficient masking strategies and traditional encoder-decoder architectures to further refine these insights.
Ultimately, the choice of bidirectionality should be driven by the intended application: bidirectional models for NLU and fine-tuning, and unidirectional models for generation and zero-shot tasks.
Summary updated