Simple and Effective Noisy Channel Modeling for Neural Machine Translation

Kyra Yee, Yann N. Dauphin, Michael Auli

Abstract: Kyra Yee, Yann Dauphin, Michael Auli. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Modern Neural Machine Translation (NMT) typically relies on "direct" models that estimate the probability of a target sentence $y$ given a source sentence $x$, denoted as $p(y|x)$. While successful, these models often struggle to leverage large-scale monolingual data effectively and can suffer from the "explaining away" effect, where the model prioritizes fluent target-side phrases over faithful translation of the source.

This paper revisits the classic "noisy channel" framework from statistical machine translation. By applying Bayes' rule, the translation probability is decomposed: $p(y|x) \propto p(x|y)p(y)$. Here, $p(x|y)$ is the channel model (translating backward) and $p(y)$ is the language model (LM). The authors demonstrate that using standard Transformer architectures within this framework yields state-of-the-art performance without the need for complex, specialized architectures.

The Noisy Channel Framework

The proposed approach utilizes three main components:

  1. Channel Model ($p(x|y)$): A standard Transformer trained to translate from the target language back to the source language. Unlike previous neural noisy channel attempts that used incremental source processing, this model scores the entire source sentence against target prefixes.
  2. Language Model ($p(y)$): A powerful Transformer LM trained on massive amounts of monolingual target-side data.
  3. Direct Model ($p(y|x)$): A standard NMT model used to prune the search space.

Decoding and Scoring

To maintain efficiency during decoding, the system uses a two-step approximation. First, the direct model proposes a small set of candidate next words. Then, the channel model and LM score these candidates. The final scoring function is a linear combination of log-probabilities normalized by length.

Standard seq2seq models (red) outperform specialized prefix-to-prefix models (black) as the target prefix length increases, justifying the use of vanilla Transformers for the channel model.

Experimental Results

The researchers evaluated the noisy channel model on several competitive benchmarks, including WMT’17 English-German (En-De), German-English (De-En), Chinese-English (Zh-En), and IWSLT’14 German-English.

Performance comparison (BLEU) across multiple datasets. The noisy channel configuration (CH+DIR+LM) consistently achieves the highest scores, outperforming direct models (DIR) and ensembles (DIR ENS).

The results indicate significant gains over strong baselines. On WMT’17 De-En, the noisy channel model improved upon the direct baseline by up to 3.2 BLEU points. Notably, the approach also outperformed ensembles of multiple direct models and "Right-to-Left" (R2L) reranking setups.

Key Insights

  • Whole Source Advantage: Scoring the entire source sentence at each decoding step is crucial, especially for language pairs with divergent word orders. Standard seq2seq models, though trained on full sentences, are remarkably robust when scoring partial target prefixes during search.
  • Principled LM Integration: The noisy channel framework provides a theoretically sound way to let a massive language model guide the translation process, outperforming simple "shallow fusion" techniques.
  • Scalability: The noisy channel approach scales better with larger beam sizes than traditional direct models. In n-best list reranking with a beam size of 100, it provided a 4.0 BLEU improvement over the baseline.

Conclusion

This work shows that the noisy channel framework, when implemented with modern Transformer architectures, remains a powerful and simple method for NMT. It provides a principled way to integrate large-scale monolingual data and improves translation quality across various language pairs and domains. While computationally more intensive than direct models, its performance gains and robustness make it a compelling choice for high-quality translation tasks.

Summary updated

BibTeX
@inproceedings{yee2019simple,
  author = {Kyra Yee and Yann N. Dauphin and Michael Auli},
  title = {{Simple and Effective Noisy Channel Modeling for Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=ZQ2_onmyVbo},
}
Figure