Introduction
In the field of Natural Language Processing (NLP), two architectural titans dominate: the Encoder-Decoder (EncDec) (e.g., T5, standard Transformer) and the Language Model (LM) (e.g., GPT family). For Machine Translation (MT), EncDec has long been the undisputed standard. However, the massive success of large-scale LMs in general text generation raises a critical question: Is the EncDec dominance in translation due to a fundamental architectural advantage, or have unified LMs simply not been scaled sufficiently for this task?
This paper investigates whether the parameter sharing inherent in a single-stack LM architecture helps or hurts translation quality. By systematically varying data conditions and model sizes, the authors examine how these models scale compared to EncDec and how effectively they transfer knowledge across languages in zero-shot scenarios.
Architectural Configurations
The researchers conducted a head-to-head comparison of three primary architectural configurations, distinguished primarily by their attention masking strategies:
- Encoder-Decoder (EncDec): The standard MT approach utilizing separate layer stacks. The source sequence is processed with bidirectional attention, while the target sequence uses causal masking.
- PrefixLM: A unified, single-stack architecture where source tokens can attend to all other source tokens (full-visible masking), but target tokens are generated autoregressively (causal masking).
- CausalLM: A "pure" LM approach where both source and target tokens are processed with strict causal masking, meaning tokens can only attend to previous tokens in the sequence.
These models were evaluated across two main axes: Scaling (increasing model size by adding layers for depth or increasing feed-forward dimensions for width) and Cross-Lingual Transfer (evaluating zero-shot translation capabilities on unseen language pairs).
Experimental Setup
To ensure robust and generalizable results, the study utilized a diverse range of benchmarks:
- Bilingual Tasks: WMT14 English-French (41M samples), WMT14 English-German (4.5M), and WMT19 English-Chinese (26M).
- Massive Scale: An in-house "Web En-De" dataset containing 2 billion samples to test the absolute limits of scaling.
- Multilingual/Zero-Shot: The OPUS-100 corpus, covering 100 languages, used to evaluate many-to-many translation and zero-shot transfer.
Key Findings
The "Scale-as-Equalizer" Effect
Architectural choices matter immensely at smaller model scales. However, as the number of parameters increases, the performance gap between EncDec and PrefixLM shrinks significantly. Inductive biases—such as the explicit separation of source and target in EncDec—become less critical when the model has sufficient capacity to learn underlying patterns directly from the data.
Furthermore, the study found that for LMs, stacking more layers (going deeper) is generally a more effective scaling strategy than simply making layers wider. Conversely, forcing the model to process the source sequence causally (as in CausalLM) consistently hurts translation quality, proving that translation fundamentally benefits from full-visible access to the source sentence.
The Zero-Shot Superpower
While EncDec maintains a slight edge on supervised tasks (offering the best balance of quality versus parameters), PrefixLM significantly outperforms EncDec in zero-shot translation.
PrefixLM solves a major bottleneck in multilingual MT known as the "off-target" problem—where a model translates into the wrong language (e.g., producing English when asked to translate German to French). On the OPUS-100 benchmark, PrefixLM achieved a 7.95 BLEU score in zero-shot directions, nearly doubling the 4.80 BLEU of the standard EncDec. Furthermore, PrefixLM's zero-shot "Language Accuracy" (producing the correct target language) was roughly 41%, compared to a mere 24% for EncDec.
Efficiency Trade-offs
Despite the impressive quality gains at scale and in zero-shot scenarios, unified LMs remain computationally more expensive. Because they concatenate the source and target into a single sequence, the quadratic computational cost of self-attention is significantly higher than in the split EncDec architecture, requiring more FLOPs for the same level of performance.
Conclusion
The EncDec architecture remains the most efficient and practical choice for specialized, high-resource language pairs or environments with strict computational constraints. However, this research demonstrates that PrefixLMs are the superior choice for massive, multilingual models where zero-shot performance and cross-lingual transfer are primary goals. Future research will likely focus on bridging the efficiency gap, aiming to make unified architectures as computationally lightweight as EncDec models without sacrificing their powerful zero-shot capabilities.
Summary updated