What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization?

Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, Colin RaffelICML 2022 · OpenReview

Introduction

In the rapidly evolving landscape of Large Language Models (LLMs), there is no single "standard" recipe for building a model. Some researchers advocate for Causal Decoders (like GPT) trained on next-token prediction, while others prefer Encoder-Decoder architectures (like T5) trained on "filling in the blanks."

Comparing these paradigms is notoriously difficult because models are typically trained on different datasets, at different scales, and for varying durations. This paper aims to level the playing field by asking a fundamental question: If we hold the compute budget constant, which combination of architecture and training objective yields the best zero-shot generalization? Furthermore, does the optimal choice change if the model undergoes a "multitask finetuning" stage after its initial pretraining?

Methodology

To answer these questions, the authors conducted a large-scale, systematic evaluation. They trained models with roughly 5 billion parameters (11 billion for the encoder-decoder to ensure compute parity) on 168 billion tokens from the C4 (Colossal Clean Crawled Corpus) dataset.

The study evaluated three architectures and three pretraining objectives:

  • Architectures: Causal Decoder (CD), Non-Causal Decoder (ND), and Encoder-Decoder (ED).
  • Objectives: Full Language Modeling (FLM), Prefix Language Modeling (PLM), and Masked Language Modeling (MLM / Span Corruption).
Overview of the experimental setup spanning architecture, objective, adaptation, and evaluation.

A key novelty in this study is the exploration of Model Adaptation. Because some models excel at open-ended generation while others are better at zero-shot tasks, the authors tested whether they could efficiently "convert" a model from one type to another. For example, they took a causal decoder and continued its training as a non-causal decoder with a masking objective to see if it could be adapted for better multitask finetuning performance.

Following pretraining (and optional adaptation), models were evaluated either directly or after Multitask Finetuning (MT-F) using the T0-Train mixture. Final zero-shot performance was measured on two major benchmarks: T0-Eval and the EleutherAI LM Evaluation Harness (EAI-Eval).

Key Insights and Results

The Multitask Finetuning "Reversal"

The most striking finding of the paper is that the "best" model architecture depends entirely on whether you plan to perform multitask finetuning.

  • Pretraining Only: If you evaluate the model straight out of pretraining, the standard GPT-style Causal Decoder with Full Language Modeling (FLM) is the clear winner. It achieved the highest average zero-shot accuracy on EAI-Eval (44.2%) and T0-Eval (42.4%).
  • After Multitask Finetuning: Once an MT-F stage is introduced, the results flip entirely. The Encoder-Decoder with Masked Language Modeling (MLM) dominates, reaching 60.6% on T0-Eval, significantly outperforming the Causal Decoder (FLM) which only reached 51.4%.

Bidirectional Attention is Key for Tasks

The success of the Encoder-Decoder and Masked Language Modeling approaches after finetuning points to the importance of attention mechanisms.

Comparison of attention patterns and pretraining objectives across the evaluated architectures.

Masked Language Modeling and Encoder-Decoder architectures allow for "non-causal" or bidirectional attention on the input sequence. This enables the model to build much richer, context-aware representations of the task instructions (the prompt) compared to a strictly causal model that can only look backward at previous tokens.

Efficient Adaptation

Fortunately, researchers do not have to choose just one paradigm from scratch. The authors demonstrated that adapting a model from one objective to another is remarkably efficient.

For instance, a Causal Decoder can be adapted into a Non-Causal Decoder with a 3.3x speedup compared to training a non-causal model from scratch. Similarly, converting a model from MLM to FLM showed a 1.6x speedup in convergence. This allows practitioners to maintain a high-quality generative model while efficiently spinning off a version specialized for zero-shot task performance.

Limitations and Future Work

While comprehensive, the authors note a few boundaries to their study:

  • Scale: At 5B-11B parameters, the models are smaller than frontier models like GPT-4 or PaLM. While the authors hypothesize these compute-efficiency findings will hold at larger scales, it remains to be empirically proven.
  • Architectural Variants: The study focused strictly on standard Transformers. Future research could explore how these findings apply to Mixture-of-Experts (MoE) architectures or models augmented with retrieval mechanisms.
  • Safety: The models were evaluated purely for research on zero-shot generalization; they were not assessed for social biases or toxicity.

Summary updated

Abstract

Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, th...
BibTeX
@inproceedings{wang2022what,
  author = {Thomas Wang and Adam Roberts and Daniel Hesslow and Teven Le Scao and Hyung Won Chung and Iz Beltagy and Julien Launay and Colin Raffel},
  title = {{What Language Model Architecture and Pretraining Objective Works Best for Zero-Shot Generalization?}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=KjiolSaKEB},
}
Figure