Standard Transformer models generate text like a scribe: one word at a time, from left to right. Each new token depends on every token that came before it. While this sequential nature makes them highly accurate, it also makes them slow. As the desired output grows longer, the time required to generate it increases linearly.
Non-autoregressive translation (NAT) aims to break this bottleneck by generating the entire sequence in parallel. Instead of $N$ sequential steps, the model attempts to produce all tokens in a single "shot," potentially speeding up inference by 10x to 25x. However, removing the step-by-step dependency introduces two fundamental challenges: the model no longer knows how long the sentence should be, and it struggles to coordinate which words go where.
The Bottleneck of Predefined Length
In an autoregressive model, the sequence ends naturally when the model decides to output an "End of Sentence" token. In a non-autoregressive setting, the model must decide the sequence length before it starts decoding.
As noted by Zhiqing Sun and Di He in Fast Structured Decoding for Sequence Models, this creates a rigid constraint. If the initial length prediction is off by even a few tokens, the resulting translation often becomes "glitchy," truncated, or repetitive. Because the decoding happens in parallel, the model lacks the flexibility to dynamically adjust the sequence length mid-generation. This dependency on an external length predictor is a recurring theme across the research, with Junliang Guo and Xu Tan emphasizing that inaccuracies here are a primary driver of quality degradation in Non-Autoregressive Neural Machine Translation with Enhanced Decoder Input.
Glancing at the Target
If you try to predict all words at once without knowing what your neighbors are choosing, you might end up with "The cat cat on the mat" because two different positions both decided "cat" was the most likely next word. To solve this "multimodality problem," models need a way to learn inter-word dependencies during training, even if they won't have access to them during inference.
The Glancing Transformer (GLAT), proposed by Hao Zhou and Lihua Qian, introduces an elegant solution: the "glancing" strategy. During training, the model is allowed to "glance" at fragments of the actual target sentence. By adaptively sampling these fragments, the model learns to fill in the remaining slots while respecting the context provided by the "glanced" tokens. This teaches the model to capture word interdependencies in a single pass, as detailed in Glancing Transformer for Non-Autoregressive Neural Machine Translation.
Bridging the Gap with Teachers
Since autoregressive (AR) models are already very good at understanding language structure, much of the research focuses on "distilling" that wisdom into non-autoregressive (NAR) models.
- Energy-Based Training: Lifu Tu and Sam Wiseman developed ENGINE, which uses a pre-trained AR teacher to define an "energy" landscape. The NAR model is trained to minimize this energy, effectively learning the teacher's entire probability distribution rather than just mimicking its top choice. This is explored in ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation.
- Curriculum Learning: Junliang Guo and Tao Qin suggest that jumping straight to parallel generation is too difficult. In Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation, they propose "soft-tuning" a model, gradually transitioning it from a sequential teacher to a parallel student.
- Syntactic Awareness: To compensate for the lack of sequential context, Jianguo Zhang and Yao Wan introduced SNAT, which explicitly feeds syntactic information like parts-of-speech and named entities into the model, as described in Enriching Non-Autoregressive Transformer with Syntactic and Semantic Structures for Neural Machine Translation.
Why This Works
The intuition behind these methods is to decouple learning from inference. During training, we use every trick available—glancing at the target, using AR teachers, or injecting syntactic structures—to help the model understand how words relate to one another.
Once the model has internalized these relationships, it can "hallucinate" the entire sentence structure simultaneously during inference. By solving the length prediction problem and using clever training objectives like those in Semi-Autoregressive Training Improves Mask-Predict Decoding by Marjan Ghazvininejad and Luke Zettlemoyer, these models achieve the holy grail of NLP: the accuracy of a sequential Transformer with the speed of a parallel processor.