Modeling Coverage for Non-Autoregressive Neural Machine Translation

Yong Shan, Yang Feng, Chenze Shao

Abstract

Non-Autoregressive Neural Machine Translation (NAT) has achieved significant inference speedup by generating all tokens simultaneously. Despite its high efficiency, NAT usually suffers from two kinds of translation errors: over-translation (e.g. repeated tokens) and under-translation (e.g. missing translations), which eventually limits the translation quality. In this paper, we argue that these issues of NAT can be addressed through coverage modeling, which has been proved to be useful in autoregressive decoding. We propose a novel Coverage-NAT to model the coverage information directly by a token-level coverage iterative refinement mechanism and a sentence-level coverage agreement, which can remind the model if a source token has been translated or not and improve the semantics consistency between the translation and the source, respectively. Experimental results on WMT14 En-De and WMT16 En-Ro translation tasks show that our method can alleviate those errors and achieve strong improvements over the baseline system.

Introduction: The Multimodality Problem in NAT

Standard Neural Machine Translation (NMT) is typically autoregressive (AT), generating text sequentially. While accurate, this word-by-word generation is a bottleneck for inference speed. Non-Autoregressive Translation (NAT) was introduced to overcome this by predicting all target tokens in parallel.

However, NAT models often struggle with the "multimodality problem." Because the model predicts every word simultaneously without explicit sequential dependencies, it can lose track of which parts of the source sentence have already been translated. This leads to two primary failure modes:

  1. Over-translation: The model repeats tokens (e.g., "the technical technical staff").
  2. Under-translation: The model omits critical words from the source (e.g., skipping "also").
Examples of over-translation (repeated tokens) and under-translation (missing words) in standard NAT.

To address these issues, this research adapts the concept of "coverage modeling"—a technique used in traditional statistical and autoregressive NMT—to the parallel architecture of NAT.

Methodology: Coverage-NAT

The proposed Coverage-NAT framework introduces two complementary mechanisms to track and enforce translation coverage at different granularities.

The Coverage-NAT architecture featuring Token-Level Coverage Iterative Refinement and Sentence-Level Coverage Agreement.

1. Token-Level Coverage Iterative Refinement (TCIR)

Unlike standard NAT decoders that perform a single pass, TCIR introduces an iterative process at the top layer of the decoder.

  • Coverage Tracking: The model maintains a coverage vector that accumulates attention weights across iterations.
  • Dynamic Biasing: In each subsequent iteration, this vector acts as a bias for the attention mechanism. It penalizes the model for attending to source tokens that have already been "covered" and encourages it to focus on remaining untranslated content.
  • Parallel Efficiency: Crucially, this refinement happens for the entire sentence at once. By using a small number of iterations (e.g., $K=5$), the model captures dependencies without sacrificing the massive speed advantages of NAT.

2. Sentence-Level Coverage Agreement (SCA)

While TCIR handles local token repetitions, SCA ensures global semantic consistency. During training, the model treats the source and generated sentences as global representations. It calculates a semantic agreement loss (L2 distance) between the source "bag-of-words" and the hypothesis expectation. This regularizer penalizes the model if the overall meaning of the translation fails to "cover" the source.

Optimization Strategy

Training is conducted in two phases to ensure stability:

  1. Pre-training: The model is trained using standard Maximum Likelihood Estimation and length prediction losses.
  2. Fine-tuning: The Sentence-Level Coverage Agreement loss is introduced to refine the model's semantic alignment.

Experimental Results

The model was evaluated on the WMT14 English-German (En-De) and WMT16 English-Romanian (En-Ro) benchmarks.

  • Translation Quality: On WMT14 En→De, Coverage-NAT achieved a significant improvement of +2.75 BLEU points over the baseline NAT system.
  • Inference Speed: Despite the iterative refinement, the model maintained a 10.04x speedup over the autoregressive Transformer.
  • Error Mitigation: Quantitative analysis confirmed a reduction in repeated tokens, particularly in longer sentences where the repetition ratio dropped from 12.05% to 7.52%.
  • Human Evaluation: Human graders noted a substantial decrease in missing translations, with Coverage-NAT scoring 70.6% on under-translation metrics compared to 58.9% for the baseline.

Discussion and Conclusion

Coverage-NAT demonstrates that the speed of non-autoregressive models does not have to come at the cost of basic translation coherence. By treating coverage as a dynamic attention bias (TCIR) and a global training objective (SCA), the model effectively mitigates the "stuttering" and "skipping" behaviors inherent in parallel generation.

Future research may look into applying coverage mechanisms across multiple decoder layers or reducing the reliance on knowledge distillation from autoregressive "teacher" models, which remains a standard but computationally expensive part of the NAT training pipeline.

Summary updated

Abstract

Non-Autoregressive Neural Machine Translation (NAT) has achieved significant inference speedup by generating all tokens simultaneously. Despite its high efficiency, NAT usually suffers from two kinds of translation errors: over-translation (e.g. repeated tokens) and under-translation (e.g. missing translations), which eventually limits the translation quality. In this paper, we argue that these issues of NAT can be addressed through coverage modeling, which has been proved to be useful in autoregressive decoding. We propose a novel Coverage-NAT to model the coverage information directly by a token-level coverage iterative refinement mechanism and a sentence-level coverage agreement, which can remind the model if a source token has been translated or not and improve the semantics consistency between the translation and the source, respectively. Experimental results on WMT14 En-De and WMT16 En-Ro translation tasks show that our method can alleviate those errors and achieve strong improvements over the baseline system.
BibTeX
@inproceedings{shan2021modeling,
  author = {Yong Shan and Yang Feng and Chenze Shao},
  title = {{Modeling Coverage for Non-Autoregressive Neural Machine Translation}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=_UZDBRmowih},
}
Figure