Abstract: Kyra Yee, Yann Dauphin, Michael Auli. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Modern Neural Machine Translation (NMT) typically relies on "direct" models that estimate the probability of a target sentence $y$ given a source sentence $x$, denoted as $p(y|x)$. While successful, these models often struggle to leverage large-scale monolingual data effectively and can suffer from the "explaining away" effect, where the model prioritizes fluent target-side phrases over faithful translation of the source.
This paper revisits the classic "noisy channel" framework from statistical machine translation. By applying Bayes' rule, the translation probability is decomposed: $p(y|x) \propto p(x|y)p(y)$. Here, $p(x|y)$ is the channel model (translating backward) and $p(y)$ is the language model (LM). The authors demonstrate that using standard Transformer architectures within this framework yields state-of-the-art performance without the need for complex, specialized architectures.
The Noisy Channel Framework
The proposed approach utilizes three main components:
- Channel Model ($p(x|y)$): A standard Transformer trained to translate from the target language back to the source language. Unlike previous neural noisy channel attempts that used incremental source processing, this model scores the entire source sentence against target prefixes.
- Language Model ($p(y)$): A powerful Transformer LM trained on massive amounts of monolingual target-side data.
- Direct Model ($p(y|x)$): A standard NMT model used to prune the search space.
Decoding and Scoring
To maintain efficiency during decoding, the system uses a two-step approximation. First, the direct model proposes a small set of candidate next words. Then, the channel model and LM score these candidates. The final scoring function is a linear combination of log-probabilities normalized by length.
Experimental Results
The researchers evaluated the noisy channel model on several competitive benchmarks, including WMT’17 English-German (En-De), German-English (De-En), Chinese-English (Zh-En), and IWSLT’14 German-English.
The results indicate significant gains over strong baselines. On WMT’17 De-En, the noisy channel model improved upon the direct baseline by up to 3.2 BLEU points. Notably, the approach also outperformed ensembles of multiple direct models and "Right-to-Left" (R2L) reranking setups.
Key Insights
- Whole Source Advantage: Scoring the entire source sentence at each decoding step is crucial, especially for language pairs with divergent word orders. Standard seq2seq models, though trained on full sentences, are remarkably robust when scoring partial target prefixes during search.
- Principled LM Integration: The noisy channel framework provides a theoretically sound way to let a massive language model guide the translation process, outperforming simple "shallow fusion" techniques.
- Scalability: The noisy channel approach scales better with larger beam sizes than traditional direct models. In n-best list reranking with a beam size of 100, it provided a 4.0 BLEU improvement over the baseline.
Conclusion
This work shows that the noisy channel framework, when implemented with modern Transformer architectures, remains a powerful and simple method for NMT. It provides a principled way to integrate large-scale monolingual data and improves translation quality across various language pairs and domains. While computationally more intensive than direct models, its performance gains and robustness make it a compelling choice for high-quality translation tasks.
Summary updated