Examining Scaling and Transfer of Language Model Architectures for Machine Translation

Biao Zhang, Behrooz Ghorbani, Ankur Bapna, Yong Cheng, Xavier Garcia, Jonathan Shen, Orhan Firat

Introduction

In the field of Natural Language Processing (NLP), two architectural titans dominate: the Encoder-Decoder (EncDec) (e.g., T5, standard Transformer) and the Language Model (LM) (e.g., GPT family). For Machine Translation (MT), EncDec has long been the undisputed standard. However, the massive success of large-scale LMs in general text generation raises a critical question: Is the EncDec dominance in translation due to a fundamental architectural advantage, or have unified LMs simply not been scaled sufficiently for this task?

This paper investigates whether the parameter sharing inherent in a single-stack LM architecture helps or hurts translation quality. By systematically varying data conditions and model sizes, the authors examine how these models scale compared to EncDec and how effectively they transfer knowledge across languages in zero-shot scenarios.

Architectural Configurations

The researchers conducted a head-to-head comparison of three primary architectural configurations, distinguished primarily by their attention masking strategies:

  1. Encoder-Decoder (EncDec): The standard MT approach utilizing separate layer stacks. The source sequence is processed with bidirectional attention, while the target sequence uses causal masking.
  2. PrefixLM: A unified, single-stack architecture where source tokens can attend to all other source tokens (full-visible masking), but target tokens are generated autoregressively (causal masking).
  3. CausalLM: A "pure" LM approach where both source and target tokens are processed with strict causal masking, meaning tokens can only attend to previous tokens in the sequence.
Attention masks for PrefixLM, CausalLM, and Encoder-Decoder architectures

These models were evaluated across two main axes: Scaling (increasing model size by adding layers for depth or increasing feed-forward dimensions for width) and Cross-Lingual Transfer (evaluating zero-shot translation capabilities on unseen language pairs).

Experimental Setup

To ensure robust and generalizable results, the study utilized a diverse range of benchmarks:

  • Bilingual Tasks: WMT14 English-French (41M samples), WMT14 English-German (4.5M), and WMT19 English-Chinese (26M).
  • Massive Scale: An in-house "Web En-De" dataset containing 2 billion samples to test the absolute limits of scaling.
  • Multilingual/Zero-Shot: The OPUS-100 corpus, covering 100 languages, used to evaluate many-to-many translation and zero-shot transfer.

Key Findings

The "Scale-as-Equalizer" Effect

Architectural choices matter immensely at smaller model scales. However, as the number of parameters increases, the performance gap between EncDec and PrefixLM shrinks significantly. Inductive biases—such as the explicit separation of source and target in EncDec—become less critical when the model has sufficient capacity to learn underlying patterns directly from the data.

Scaling behavior of different architectures on the En-Fr translation task

Furthermore, the study found that for LMs, stacking more layers (going deeper) is generally a more effective scaling strategy than simply making layers wider. Conversely, forcing the model to process the source sequence causally (as in CausalLM) consistently hurts translation quality, proving that translation fundamentally benefits from full-visible access to the source sentence.

The Zero-Shot Superpower

While EncDec maintains a slight edge on supervised tasks (offering the best balance of quality versus parameters), PrefixLM significantly outperforms EncDec in zero-shot translation.

PrefixLM solves a major bottleneck in multilingual MT known as the "off-target" problem—where a model translates into the wrong language (e.g., producing English when asked to translate German to French). On the OPUS-100 benchmark, PrefixLM achieved a 7.95 BLEU score in zero-shot directions, nearly doubling the 4.80 BLEU of the standard EncDec. Furthermore, PrefixLM's zero-shot "Language Accuracy" (producing the correct target language) was roughly 41%, compared to a mere 24% for EncDec.

Efficiency Trade-offs

Despite the impressive quality gains at scale and in zero-shot scenarios, unified LMs remain computationally more expensive. Because they concatenate the source and target into a single sequence, the quadratic computational cost of self-attention is significantly higher than in the split EncDec architecture, requiring more FLOPs for the same level of performance.

Conclusion

The EncDec architecture remains the most efficient and practical choice for specialized, high-resource language pairs or environments with strict computational constraints. However, this research demonstrates that PrefixLMs are the superior choice for massive, multilingual models where zero-shot performance and cross-lingual transfer are primary goals. Future research will likely focus on bridging the efficiency gap, aiming to make unified architectures as computationally lightweight as EncDec models without sacrificing their powerful zero-shot capabilities.

Summary updated

Abstract

Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing. In machine translation, EncDec has long been the favoured approach, but with few studies investigating the performance of LMs. In this work, we thoroughly examine the role of several architectural design choices on the performance of LMs on bilingual, (massively) multilingual and zero-shot translation tasks, under systematic variations of data conditions and model sizes. Our results show that: (i) Different LMs have different scaling properties, where architectural differences often have a significant impact on model performance at small scales, but the performance gap narrows as the number of parameters increases, (ii) Several design choices, including causal masking and language-modeling objectives for the source sequence, have detrimental effects on translation quality, and (iii) When paired with full-visible masking for source sequences, LMs could perform on par with EncDec on supervised bilingual and multilingual translation tasks, but improve greatly on zero-shot directions by facilitating the reduction of off-target translations.
BibTeX
@inproceedings{zhang2022examining,
  author = {Biao Zhang and Behrooz Ghorbani and Ankur Bapna and Yong Cheng and Xavier Garcia and Jonathan Shen and Orhan Firat},
  title = {{Examining Scaling and Transfer of Language Model Architectures for Machine Translation}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=PlFtf_pnkZu},
}
Figure