Introduction
Neural Machine Translation (NMT) has traditionally relied on autoregressive (AR) models, such as the standard Transformer. These models generate sequences token-by-token, where each prediction is conditioned on all previously generated tokens. While this sequential dependency ensures high linguistic coherence, it creates a significant computational bottleneck during inference. Non-Autoregressive Translation (NAT) emerged as a solution to this bottleneck, aiming to generate all tokens in parallel to achieve massive speedups.
The Multi-Modality Challenge
The primary obstacle for NAT is the "multi-modality problem." In AR models, the conditional dependency naturally handles cases where multiple valid translations exist. In contrast, because NAT models predict tokens independently, they often struggle to capture internal target-side dependencies. For instance, translating "Thank you" into German could result in "Vielen Dank" or "Danke." Without coordination between token slots, an NAT model might erroneously combine these into "Danke Dank." Solving this failure to capture target-side dependency is the central theme of NAT research.
A Taxonomy of NAT Methods
To overcome the multi-modality problem, researchers have developed a variety of strategies categorized into five main pillars: data manipulation, modeling, training criteria, decoding strategies, and pre-training.
- Data Manipulation: Knowledge Distillation (KD) is the most impactful technique. By training NAT models on the output of an AR teacher model, the target distribution becomes more deterministic and easier for the NAT model to learn.
- Modeling: This includes Iteration-based models (e.g., CMLM, Levenshtein Transformer) that refine outputs over several passes, and Latent Variable-based models that use hidden structures (like "fertility") to guide generation.
- Training Criteria: Alternatives to standard Cross-Entropy, such as Connectionist Temporal Classification (CTC) or Order-Agnostic Cross-Entropy (OAXE), allow for flexibility in word ordering during training.
- Decoding Strategies: Since NAT models lack an end-of-sentence token, they utilize explicit length prediction modules to determine the output sequence length.
- Pre-training: Initializing NAT architectures with large-scale pre-trained models like BERT or XLM-R significantly improves their linguistic grounding.
The Evolution of NAT Performance
Since the introduction of the first NAT model in late 2017, the field has seen rapid progress. Early models suffered from a significant accuracy gap compared to AR counterparts, often lagging by as much as 10 BLEU points.
Modern architectures like the DA-Transformer and GLAT have narrowed this gap to within 1 BLEU point on standard benchmarks like WMT14 English-German. These advancements have been achieved while maintaining impressive inference speedups, typically ranging from 15x to 30x compared to standard Transformers at batch size 1. However, it is important to note that NAT's speed advantage is most pronounced in real-time (low batch size) scenarios; as batch sizes increase, the relative efficiency gain over AR models decreases.
Future Directions
Despite the success of NAT in translation, several challenges remain:
- Reducing KD Dependency: Most high-performing NAT models still require an AR teacher. Developing effective methods to train NAT models directly on raw data remains a major goal.
- Scaling to Large Language Models (LLMs): While NAT is well-studied for base-sized models, its application to multi-billion parameter LLMs is an open research area.
- Creative Generation: NAT is currently most effective for structured tasks like translation or speech recognition. Its performance in open-ended or creative tasks, such as long-form dialogue, requires further exploration.
Summary updated