In sequence generation tasks like machine translation, automatic speech recognition (ASR), and image captioning, the standard recipe has long been autoregressive decoding. Autoregressive (AR) models generate text token by token, where each predicted token is conditioned on all previously generated tokens. Mathematically, this leverages the chain rule of probability to perfectly capture step-by-step linguistic dependencies.
However, this sequential nature imposes a strict computational bottleneck: generating a sequence of length $N$ requires $N$ sequential forward passes through the network, resulting in $O(N)$ inference latency.
To break this bottleneck, researchers introduced non-autoregressive (NAR) generation. As pioneered by Jiatao Gu et al. in their work on Non-Autoregressive Neural Machine Translation, NAR models predict all output tokens simultaneously in a single parallel pass ($O(1)$ latency). While this achieves massive inference speedups—often between 5x and 50x—it introduces a significant performance deficit known as the non-autoregressive quality gap.
Understanding why this gap occurs, and how modern architectures systematically close it, is central to the evolution of efficient sequence modeling.
The Root of the Gap: The Multimodality Problem
The performance gap stems from a fundamental mathematical assumption: conditional independence. By generating all tokens in parallel, a vanilla NAR model assumes that the probability of emitting any token $y_t$ is entirely independent of its neighboring token $y_{t-1}$, given the source context $X$.
When the target distribution is highly multimodal—meaning there are multiple valid ways to express the same idea—this independence assumption fails catastrophically.
Consider translating a phrase into a language with multiple valid word orders or synonyms. An AR model picks one token, and then is strictly bound to follow that specific linguistic path. An NAR model, lacking an internal language model to coordinate its parallel predictions, attempts to optimize for all valid paths simultaneously. As Vincent Xie et al. note, parallel architectures frequently fail to model the relationships between tokens guessed at the exact same time.
The result is severe structural incoherence. If pushed to extreme speedup ratios without correction, NAR models produce distinct linguistic artifacts, such as repetitive phrasing, omitted words, and grammatical chaos, as documented by Shikhar Tuli et al.
To close the performance gap without reintroducing the $O(N)$ sequential bottleneck, researchers have developed three primary paradigms: iterative refinement, hybrid semi-autoregressive decoding, and cross-model knowledge distillation.
Strategy 1: Iterative Refinement and Imputation
Instead of forcing the model to generate the entire sequence perfectly in a single shot, iterative refinement frameworks allow the model to generate text over a small, fixed number of steps. This maintains a constant time complexity independent of the sequence length.
Step 1 (Pure NAR): [Vielen] [schön] [ ] [ ] (Incoherent mix)
Step 2 (Masked): [Vielen] [Dank] [ ] [ ] (Self-corrected)
Step 3 (Final): [Vielen] [Dank] [für] [Alles] (Coherent output)
A prominent example of this is the Imputer, introduced by Mohammad Norouzi et al. The Imputer generates sequences by performing "imputations"—filling in blanks on a fixed-size canvas. By using dynamic programming to marginalize over latent alignments, it systematically resolves token dependencies across a small number of decoding passes.
In the domain of speech recognition, Yosuke Higuchi et al. designed an improved Mask-CTC framework. This architecture uses a parallel model to generate an initial token hypothesis, masks out tokens with low confidence scores, and then re-predicts them conditioned on the high-confidence tokens. This iterative loop allows the model to dynamically fix its own insertion and deletion errors, narrowing the accuracy gap with sequential models while preserving a 5x speedup.
Strategy 2: Hybrid and Semi-Autoregressive Decoding
Another highly effective approach is to introduce coarse-grained or localized autoregression, striking a deliberate balance between global sequence structure and local generation speed.
Lukasz Kaiser et al. bypassed the multimodality problem by introducing discrete latent variables. Their model first autoregressively predicts a highly compressed, short sequence of latent codes that capture the global meaning and structure of the sentence. Once this latent backbone is established, the full, high-resolution text sequence is decoded completely in parallel. Because the global choices are made sequentially, the parallel decoder is no longer susceptible to mixing conflicting translation paths.
For vision-language tasks like image captioning, Zhengcong Fei proposed a partially non-autoregressive framework. Instead of treating the entire caption as a single parallel block, the sequence is broken into groups. The model generates these groups of words in parallel to maximize throughput, but maintains strict autoregressive logic within each group to preserve local grammatical dependencies and fine-grained visual nuances.
Strategy 3: Knowledge Distillation and Joint Training
Rather than changing the decoding mechanics, some approaches focus on altering the data distribution itself during training to make parallel prediction easier.
The multimodality of natural language is highly complex. However, the output distribution of a trained autoregressive model is significantly smoother and less ambiguous than raw human text. By training an NAR model on the outputs of an AR "teacher" model—a process known as sequence-level knowledge distillation—the NAR model is exposed to a simplified target space where synonyms and word orders are highly consistent.
Xiaobo Liang et al. formalized this synergy through JANUS, a framework that enforces joint autoregressive and non-autoregressive training. By introducing a shared auxiliary distribution, the two architectures mutually regularize one another, transferring the linguistic fluency of sequential prediction directly into the parallel weights of the NAR model.
Similarly, in speech processing, Xun Gong demonstrated that combining frame-level and sequence-level distillation from an AR teacher allows compact NAR models to retain high word accuracy, proving that the structural intelligence of sequential models can be effectively baked directly into parallel architectures.