Introduction
Neural Machine Translation (NMT) has traditionally relied on large parallel corpora. However, the scarcity of such "bitext" has led researchers to leverage massive monolingual datasets. Two dominant paradigms have emerged: Back-translation (BT), which generates synthetic parallel data, and Pre-training (BERT), which incorporates deep linguistic representations from models like BERT into the NMT architecture. This paper provides a comprehensive head-to-head comparison, evaluating whether the computational cost of BERT-fused models translates to superior performance over the simpler, more efficient BT method.
Methodology
The study compares two specific implementations:
- BERT-fused NMT: This architecture uses BERT as an auxiliary memory module. The NMT encoder and decoder interact with BERT's representations through an additional attention mechanism, allowing the model to "consult" BERT during translation.
- Large-scale Back-translation: A standard Transformer model is trained on a mix of authentic bitext and synthetic data generated by translating target-language monolingual text back into the source language. The authors explore various ratios of authentic-to-synthetic data, ranging from 1:0.5 to 1:4.
Both methods utilize the same monolingual data source (Wikipedia) to ensure a fair comparison of their ability to extract useful information from non-parallel text.
Experimental Setup
The models were evaluated on two distinct translation tasks:
- En→De (WMT’14): A high-resource scenario with 4.5 million sentence pairs.
- Zh→En (LDC): A lower-resource scenario with 1.25 million sentence pairs.
Beyond standard BLEU scores, the authors employed eight specialized test suites to probe specific linguistic capabilities, including morphology, syntax, semantics (homographs, conjunctions, pronouns), and pragmatics (idioms, reasoning, gender bias).
Performance and Efficiency
The results indicate that while BERT is a powerful tool, it does not consistently outperform back-translation in terms of general translation quality.
As shown in the table above, the BT models achieved higher BLEU scores in both language pairs. For English-to-German, the BT (1:0.5) model reached 30.41, surpassing the BERT-fused model's 30.03. In the Chinese-to-English task, the BT (1:1) model peaked at 47.23, compared to 46.55 for the BERT-fused approach.
Furthermore, the efficiency gap is significant. In the En→De task, the BT model was approximately 3.6 times faster at decoding than the BERT-fused model. The latter's reliance on a massive pre-trained module introduces substantial parameter overhead and computational latency.
Linguistic Specialization: Semantics vs. Morphology
The most striking findings emerge from the fine-grained linguistic analysis. BERT proves to be a "Semantic Specialist." It excels in tasks requiring deep contextual understanding, such as resolving ambiguous pronouns or translating idioms.
As illustrated in the bar chart, the BERT-fused model (green) consistently outperforms all back-translation variants across lexical and syntactic ambiguity types. This suggests that BERT captures global topical properties and complex semantic relations that synthetic data generation fails to replicate.
Conversely, back-translation acts as a "Morphological Workhorse." It often matches or exceeds BERT's performance in handling complex grammar and word forms. The authors hypothesize that NMT morphological rules are fundamentally bilingual mapping problems, which BT addresses directly by providing more mapping examples, whereas BERT's monolingual training is less optimized for this specific requirement.
Social Bias and Pragmatics
A critical observation regarding pre-training is the introduction of "inductive social bias." The BERT-fused model exhibited significantly higher gender bias, struggling with anti-stereotypical sentences. In contrast, the BT models maintained a better balance between accuracy and neutrality. For instance, the BT (1:2) model achieved 75.1% accuracy with a bias score (ΔS) of 5.2, while the BERT-fused model's bias score was nearly triple at 14.6.
Conclusion
The choice between pre-training and back-translation involves a trade-off between semantic depth and computational efficiency. While BERT-fused models are superior for complex semantic tasks like idiom translation and coreference resolution, back-translation remains the more effective and efficient choice for general translation quality and grammatical accuracy.
Summary updated