Abstract
Existing approaches to neural machine translation condition each output word on previously generated outputs. We introduce a model that avoids this autoregressive property and produces its outputs in parallel, allowing an order of magnitude lower latency during inference. Through knowledge distillation, the use of input token fertilities as a latent variable, and policy gradient fine-tuning, we achieve this at a cost of as little as 2.0 BLEU points relative to the autoregressive Transformer network used as a teacher. We demonstrate substantial cumulative improvements associated with each of the three aspects of our training strategy, and validate our approach on IWSLT 2016 English-German and two WMT language pairs. By sampling fertilities in parallel at inference time, our non-autoregressive model achieves near-state-of-the-art performance of 29.8 BLEU on WMT 2016 English-Romanian.
Introduction
Most modern Neural Machine Translation (NMT) models, including the standard Transformer, are autoregressive. This means they generate text one token at a time, where each new word depends on all the words generated before it. While effective, this creates a massive computational bottleneck: the $n$-th word cannot be computed until the $(n-1)$-th word is finished. This sequential dependency limits the potential for parallelization during inference.
The authors identify the "multimodality problem" as the primary obstacle to parallel generation. If a model predicts every word independently (non-autoregressively), it might produce incoherent "mix-and-match" results. For example, if "Thank you" can be translated into German as either "Danke" or "Vielen Dank", an independent model might accidentally output "Vielen" (from the second option) and then stop, or produce "Danke Dank."
The Non-Autoregressive Transformer (NAT)
The NAT architecture builds on the Transformer but removes the "causal mask" that forces the decoder to look only at the past. To enable coherent parallel generation, the model introduces a novel fertility predictor.
The fertility predictor adds a module that assigns a "fertility" value to each input word—an integer representing how many output words that input word will produce. These fertilities act as a "latent plan," telling the decoder exactly how long the output should be and which parts of the source sentence to focus on for each output slot. This provides the global coordination that non-autoregressive models typically lack.
Training Strategies
To make the NAT effective, the authors introduce three key innovations in the training pipeline:
- Sequence-Level Knowledge Distillation: Instead of training on raw human translations, which are often multimodal and noisy, the NAT is trained on the output of a pre-trained autoregressive Transformer (the "teacher"). This distilled data is more deterministic and easier for a non-autoregressive model to learn.
- Policy Gradient Fine-Tuning: Because fertility prediction is a discrete operation, the authors use reinforcement learning (specifically the REINFORCE algorithm) to fine-tune the model, optimizing it against the teacher’s distribution.
- Noisy Parallel Decoding (NPD): At inference time, the model can sample multiple fertility sequences in parallel. An autoregressive teacher then quickly scores these candidates to select the best translation. This "best-of-many" approach significantly boosts accuracy with minimal impact on latency.
Experimental Results
The authors evaluated the NAT on three standard benchmarks: IWSLT16 English–German, WMT14 English–German, and WMT16 English–Romanian.
As shown in the results, the NAT achieves a massive 15.6x speedup over a greedy autoregressive Transformer. On WMT16 English-Romanian, the NAT reached 29.8 BLEU, trailing the state-of-the-art autoregressive model by only a small margin. Unlike standard models, where latency grows linearly with sentence length, the NAT’s latency remains nearly constant for typical sentence lengths, as the decoder slots are processed in parallel.
Conclusion
The Non-Autoregressive Transformer successfully breaks the sequential bottleneck of traditional NMT. By utilizing fertility as a latent variable and leveraging knowledge distillation, the NAT achieves an order of magnitude reduction in latency while maintaining competitive translation quality. While a small BLEU gap remains on more complex datasets, the NAT represents a significant step toward real-time, high-quality machine translation.
Summary updated