TL;DR
This paper introduces Coverage-NAT, a novel approach to fix the common "stuttering" (over-translation) and "skipping" (under-translation) errors in fast, non-autoregressive translation models. By implementing iterative refinement at the token level and a semantic agreement loss at the sentence level, the authors achieve significant quality improvements (up to +2.75 BLEU) while remaining roughly 10x faster than traditional models.
What Problem Does This Paper Address?
Standard Neural Machine Translation (NMT) is autoregressive (AT), meaning it generates text one word at a time. While accurate, this is slow. Non-Autoregressive Translation (NAT) was proposed to speed things up by generating all tokens simultaneously.
However, NAT suffers from the "multimodality problem." Because the model predicts every word at once without knowing what the other positions are doing, it lacks sequential dependency. This often results in two types of errors:
- Over-translation: The model gets "stuck" and repeats the same token (e.g., "the technical technical staff").
- Under-translation: The model ignores parts of the source sentence entirely (e.g., skipping the word "also").
In older translation systems, "coverage modeling" was used to track which source words had been translated. This paper adapts that concept for the parallel world of NAT.
Methods
The authors propose two main mechanisms under the Coverage-NAT framework:
1. Token-Level Coverage Iterative Refinement (TCIR)
In a standard NAT decoder, all steps happen at once. TCIR replaces the top layer of the decoder with an iterative process.
- Tracking: It maintains a "coverage vector" that accumulates attention weights from previous iterations.
- Biasing: In the next iteration, the model uses this vector as a bias. It discourages the decoder from attending to source tokens that were already "covered" and pushes it to focus on untranslated parts.
- Efficiency: Because this refinement happens across the whole sentence in parallel (iteration-wise) rather than word-by-word (step-wise), it preserves the speed advantage of NAT.
2. Sentence-Level Coverage Agreement (SCA)
This is a regularization term used during training. It treats the entire source and generated sentences as "bags-of-words" and calculates a global semantic representation for each. The model is then penalized (via L2 distance) if the generated sentence's meaning doesn't "cover" the source sentence's meaning.
Two-Phase Optimization
The authors found that training with all losses at once was unstable. They use a two-phase approach: first, pre-train the model with standard translation losses, then fine-tune it using the new Sentence-Level Coverage Agreement.
Datasets
The model was evaluated on two standard benchmarks:
- WMT14 En↔De: A large-scale dataset with 4.5 million sentence pairs.
- WMT16 En↔Ro: A smaller dataset with 610k sentence pairs. The authors used the standard Transformer model as the "teacher" for knowledge distillation, which is a common practice to simplify the target distribution for NAT models.
Key Insights and Ideas
- Iterative is better than Sequential for NAT: You don't need to generate word-by-word to have "memory." By refining the entire sentence over a few iterations, you can model dependencies without the massive latency of autoregressive decoding.
- Coverage as an Attention Bias: The "aha moment" here is treating coverage not just as a metric, but as a dynamic bias that reshapes the attention mechanism in real-time during inference.
- Global vs. Local: Over-translation is often a local error, while under-translation is a global semantic error. By tackling both token-level (TCIR) and sentence-level (SCA) coverage, the authors address both sides of the multimodality coin.
Key Results
- Quality Gains: On WMT14 En→De, Coverage-NAT improved the baseline by +2.75 BLEU points.
- Speed: The model achieved a 10.04x speedup over the standard autoregressive Transformer. Even with "rescoring" (a technique to pick the best of several candidates), it remained over 5x faster while nearly matching the Transformer's accuracy.
- Error Reduction: Quantitative analysis showed that the ratio of repeated tokens dropped significantly, especially in long sentences (from 12.05% down to 7.52%).
- Human Evaluation: In subjective tests, humans found that Coverage-NAT significantly reduced missing translations, scoring 70.6% on the "under-translation metric" compared to the baseline's 58.9%.
Limitations and Future Work
- Iteration Trade-off: While 5 iterations (the paper's default) provide a good balance, increasing iterations further improves BLEU but begins to eat into the speed advantage.
- Teacher Dependency: Like most NAT research, this model relies heavily on a pre-trained AT "teacher" model for distillation. Future work might explore how to achieve these coverage gains without needing a slow teacher model during the training phase.
- Top-Layer Focus: The TCIR mechanism is currently only applied to the top layer of the decoder; exploring multi-layer coverage integration could potentially yield even higher consistency.
Summary updated