Abstract: Natural language understanding and generation models follow one of the two dominant architectural paradigms: language models (LMs) that process concatenated sequences in a single stack of layers, and encoder-decoder models (EncDec) that utilize separate layer stacks for input and output processing. In machine translation, EncDec has long been the favoured approach, but with few studies investigating the performance of LMs. In this work, we thoroughly examine the role of several architectural design choices on the performance of LMs on bilingual, (massively) multilingual and zero-shot translation tasks, under systematic variations of data conditions and model sizes. Our results show that: (i) Different LMs have different scaling properties, where architectural differences often have a significant impact on model performance at small scales, but the performance gap narrows as the number of parameters increases, (ii) Several design choices, including causal masking and language-modeling objectives for the source sequence, have detrimental effects on translation quality, and (iii) When paired with full-visible masking for source sequences, LMs could perform on par with EncDec on supervised bilingual and multilingual translation tasks, and improve greatly on zero-shot directions by facilitating the reduction of off-target translations.
Introduction
In the field of Natural Language Processing, two primary architectural paradigms dominate: Encoder-Decoder (EncDec) models, which utilize separate stacks for input and output, and unified Language Models (LMs), which process sequences in a single stack. While EncDec has been the standard for machine translation (MT) due to its explicit separation of source understanding and target generation, the massive scaling success of LMs like GPT raises a critical question: can unified LMs match or exceed EncDec performance in translation tasks as they scale?
This paper investigates how architectural design choices—specifically masking strategies and layer-wise encoding—affect MT performance across bilingual, multilingual, and zero-shot scenarios. The study focuses on two key dimensions: scaling (how performance improves with parameter count) and cross-lingual transfer (the ability to translate between language pairs not explicitly seen during training).
Architectural Variants and Methodology
The researchers compared several architectural configurations to isolate the impact of specific design choices:
- EncDec: The traditional baseline with separate encoder and decoder stacks.
- CausalLM: A standard unidirectional LM that processes both source and target with a strict left-to-right (causal) mask.
- PrefixLM: A unified model where the source sequence uses bidirectional (full-visibility) masking, while the target remains causal. This allows the model to look at the entire source sentence simultaneously, similar to an encoder.
- TopOnly vs. Layer-Wise Encoding: A test of whether the generation process should attend to every layer of the source representation (standard LM) or only the final layer (standard EncDec).
These models were evaluated across diverse benchmarks, including WMT14 (En-De, En-Fr), WMT19 (En-Zh), and the massively multilingual OPUS-100 dataset. To test scaling limits, the authors also utilized a large-scale "Web En-De" dataset containing 2 billion samples.
Empirical Results and Scaling Laws
Scaling as the Great Equalizer
A primary finding is that architectural differences are most pronounced at smaller scales. As models grow toward billions of parameters, the performance gap between PrefixLM and EncDec narrows significantly. Both architectures follow predictable power-law scaling: $L(N) = \alpha(N_0/N)^p + L_\infty$, where the scaling exponent ($p$) remains remarkably consistent across different designs.
The Impact of Masking
The study confirms that causal masking on the source sequence is detrimental to translation quality. Translation requires a holistic, bidirectional understanding of the source; forcing a model to process the source unidirectionally (as in CausalLM) consistently hurts BLEU scores compared to PrefixLM or EncDec.
Zero-Shot Breakthroughs
Perhaps the most significant discovery is in zero-shot translation. While EncDec maintains a slight edge in supervised tasks, PrefixLM is vastly superior in zero-shot scenarios. On the OPUS-100 zero-shot benchmark, PrefixLM achieved BLEU scores of 7.44-7.95, nearly doubling the 4.80 achieved by EncDec. This is largely due to a reduction in "off-target" errors; PrefixLMs reached ~41% language accuracy compared to EncDec's 24%, suggesting that sharing parameters in a single stack helps the model maintain better cross-lingual consistency.
Discussion and Limitations
While PrefixLMs show promise, they face efficiency challenges. Because they treat source and target as a single sequence, the quadratic cost of self-attention becomes more expensive as sequence length increases compared to the split attention of EncDec. Furthermore, in low-data regimes, the inductive biases of the EncDec architecture remain highly beneficial. Future work is needed to reconcile the zero-shot advantages of unified LMs with the computational efficiency of specialized encoder-decoder structures.
Summary updated