Simple and Effective Noisy Channel Modeling for Neural Machine Translation

Kyra Yee, Nathan Ng, Yann N. Dauphin, Michael Auli

Abstract: Previous work on neural noisy channel modeling relied on latent variable models that incrementally process the source and target sentence. This makes decoding decisions based on partial source prefixes even though the full source is available. We pursue an alternative approach based on standard sequence to sequence models which utilize the entire source. These models perform remarkably well as channel models, even though they have neither been trained on, nor designed to factor over incomplete target sentences. Experiments with neural language models trained on billions of words show that noisy channel models can outperform a direct model by up to 3.2 BLEU on WMT'17 German-English translation. We evaluate on four language-pairs and our channel models consistently outperform strong alternatives such right-to-left reranking models and ensembles of direct models.

Introduction

Modern Neural Machine Translation (NMT) has largely converged on "direct" models that estimate the conditional probability of a target sentence $y$ given a source sentence $x$, denoted as $p(y|x)$. While highly effective, these models often struggle to fully exploit massive monolingual datasets and can suffer from "explaining away," where the model prioritizes fluent, common target prefixes over accurate translation of the source content.

This paper revisits the classic noisy channel model from statistical machine translation, reformulating it for the Transformer era. By applying Bayes' Rule, the translation problem is decomposed: $p(y|x) \propto p(x|y)p(y)$. This allows for a dedicated channel model $p(x|y)$ to ensure faithfulness to the source and a language model $p(y)$ to ensure fluency, the latter of which can be trained on vast amounts of unpaired target-language text.

Methodology

The proposed approach is characterized by its simplicity. Instead of complex latent variable architectures, the authors use standard Transformer models for all components.

  1. Channel Model ($p(x|y)$): A standard seq2seq model trained to translate in the reverse direction (target to source).
  2. Language Model ($p(y)$): A standard Transformer language model trained on billions of words of monolingual data.
  3. Two-Step Decoding: To manage the computational cost of scoring with multiple models, a "pruning" strategy is employed. A direct model $p(y|x)$ first proposes a small set of candidate next words (e.g., the top $k$). The channel and language models then re-score only these candidates.
  4. Global Source Context: Unlike previous incremental approaches, this model utilizes the entire source sequence $x$ to score every partial target prefix $y_{<t}$.

Robustness of Standard Models

A key finding is that standard seq2seq models, despite being trained on complete sentence pairs, are remarkably robust when used to score partial target prefixes during decoding.

Figure 1: Comparison of a standard seq2seq model and a model specifically trained on prefixes. The standard model is robust and eventually superior.

As shown in Figure 1, while a model specifically trained on prefixes might have a slight edge for very short sequences, the standard seq2seq model (the "channel model") quickly catches up and eventually achieves higher BLEU scores as the target prefix length increases. This justifies the "simplicity" of the approach: specialized architectures for prefix-processing are unnecessary.

Main Results

The authors evaluated the noisy channel approach across several competitive benchmarks, including WMT’17 (En-De, De-En, Zh-En) and IWSLT’14 (De-En).

Figure 6: BLEU scores across four language pairs, comparing the noisy channel model (CH+DIR+LM) against direct baselines and ensembles.

The results in Figure 6 demonstrate consistent and significant improvements. On the WMT’17 En-De task, the noisy channel model outperformed the direct baseline by 3.2 BLEU points. Crucially, it also outperformed strong alternatives such as ensembles of direct models (DIR ENS) and right-to-left reranking models.

Conclusion

The noisy channel framework provides a powerful way to integrate large-scale monolingual data and improve translation faithfulness. By showing that standard Transformers can serve as effective channel models without specialized training, this work simplifies the implementation of noisy channel NMT. While the approach introduces additional computational overhead during decoding, the substantial gains in translation quality make it a compelling alternative to standard direct models.

Summary updated

Abstract

Previous work on neural noisy channel modeling relied on latent variable models that incrementally process the source and target sentence. This makes decoding decisions based on partial source prefixes even though the full source is available. We pursue an alternative approach based on standard sequence to sequence models which utilize the entire source. These models perform remarkably well as channel models, even though they have neither been trained on, nor designed to factor over incomplete target sentences. Experiments with neural language models trained on billions of words show that noisy channel models can outperform a direct model by up to 3.2 BLEU on WMT'17 German-English translation. We evaluate on four language-pairs and our channel models consistently outperform strong alternatives such right-to-left reranking models and ensembles of direct models.
BibTeX
@inproceedings{yee2019simple,
  author = {Kyra Yee and Nathan Ng and Yann N. Dauphin and Michael Auli},
  title = {{Simple and Effective Noisy Channel Modeling for Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=ZQ2_onmyVbo},
}
Figure