Standard language models generate text the way humans write: one token at a time, from left to right. This autoregressive approach ensures that every new word is conditioned on all previously generated words. Mathematically, the probability of a sequence is modeled as:
$$P(X) = \prod_{t=1}^{T} P(x_t \mid x_{<t})$$
While this sequential conditioning guarantees high coherence, it creates a massive computational bottleneck during inference because a model must run sequentially for hundreds of steps.
To unlock massive speedups, non-autoregressive (NAR) models attempt to break this bottleneck by generating all tokens simultaneously in a single forward pass. They drop the step-by-step history and assume that, given the source input, every target token position is conditionally independent of the others:
$$P(X) = \prod_{t=1}^{T} P(x_t \mid \text{Source})$$
While this parallel approach can speed up inference by up to 15x, it introduces a fundamental flaw known as the multimodality problem.
The Intuition Behind the Multimodality Problem
The inherent flexibility of language means there are often many correct ways to translate a sentence or express an idea. This creates a high-entropy target space with multiple valid paths, or "modes."
Imagine asking a parallel model to translate a phrase that has two equally valid translations: "Thank you very much" and "Many thanks to you".
An autoregressive model picks the first word of the first option ("Thank"), and then, strictly conditioned on that choice, is guided down the rest of that specific linguistic path.
An NAR model, however, predicts every position independently and simultaneously. Slot 1 sees that both "Thank" and "Many" are highly probable. Slot 2 sees that both "you" and "thanks" are highly probable. Because the slots cannot communicate or coordinate, the model might simultaneously choose "Thank" for Slot 1 and "thanks" for Slot 2, resulting in a garbled, incoherent mixture like "Thank thanks very to you".
As noted by Jiatao Gu et al. in their foundational paper on Non-Autoregressive Neural Machine Translation, this lack of coordination causes parallel generation models to inconsistently combine fragments of mutually exclusive target sequences.
Mechanics of Parallel Incoherence
When a model is forced to navigate a multimodal distribution without sequential dependencies, the token coordination failure manifests in distinct structural errors:
- Token Repetition and Omission: Because individual output slots cannot track what their neighbors are doing, they often default to redundant, high-probability local patterns to minimize loss against a static reference. As Jongyoon Song explains in the development of AligNART, this lack of structural guidance prevents the model from tracking which parts of the source have already been translated, leading to severe stuttering or skipped words.
- Syntactic Blending: In languages with flexible word orders, the target space contains both short-range and long-range structural variations. Kexun Zhang et al. demonstrate in A Study of Syntactic Multi-Modality in Non-Autoregressive Machine Translation that when an NAR model attempts to average these distinct syntactic modes during training, it produces invalid, mixed sentence structures at inference time.
- Context Mismatches: Even in semi-parallel systems that predict small groups of tokens at once, generating later tokens without knowing the exact realization of preceding tokens introduces a context mismatch, resulting in statistical inconsistencies and reduced global coherence (Tan Dat Nguyen et al. in Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding).
How the Field Mitigates Multimodality
To preserve the speed of parallel decoding without sacrificing coherence, researchers have developed three primary strategies: data simplification, latent coordination spaces, and iterative refinement.
1. Data Simplification via Knowledge Distillation
If the root cause of the problem is that the data contains too many valid modes, the most direct solution is to make the data unimodal. This is achieved through Knowledge Distillation (KD), where a parallel model is trained on the outputs of an autoregressive teacher model rather than raw human data.
As explored by the authors of Understanding Knowledge Distillation in Non-autoregressive Machine Translation, KD acts as a powerful complexity filter. Weijia Xu et al. show in How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation? that the teacher model provides simplified word orders and reduced lexical variety. By stripping away competing modes, KD allows the parallel student model to safely converge on a single, deterministic translation path.
To prevent this simplification from hurting translation diversity, Chenze Shao et al. developed One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation. This approach provides the parallel model with multiple high-quality teacher options and allows it to dynamically learn from the specific reference that best matches its own internal predictions.
2. Latent Variable Planning
Instead of predicting tokens blindly, a model can first generate a low-dimensional "plan" or structural anchor that coordinates the subsequent parallel generation.
In early work, Jiatao Gu et al. introduced "fertility" as a latent variable to predict how many target words each source word should generate, anchoring the sentence length and layout. Moving beyond discrete counts, Raphael Shu et al. introduced LaNMT, which uses continuous latent variables to iteratively refine a low-dimensional structural plan before decoding the full sequence.
Similarly, Lukasz Kaiser et al. demonstrated in Fast Decoding in Sequence Models Using Discrete Latent Variables that autoencoding sequences into a much shorter sequence of discrete latent codes allows a model to resolve global structural choices first, ensuring that the final parallel token generation remains globally consistent.
3. Iterative Refinement and Error Correction
If full parallelism destroys coherence, a natural compromise is to generate text in a few coordinated parallel steps rather than all at once.
Omer Levy introduced Mask-Predict, a conditional masked language model that generates an entire sequence in parallel, identifies low-confidence tokens, masks them out, and re-predicts them conditioned on the high-confidence tokens. This multi-step parallel decoding reintroduces conditional context, allowing the model to resolve contradictions over a few iterations.
Iteration 1 (Full Parallel): [Thank] [thanks] [very] [much] (Incoherent mixture)
Confidence Score: (High) (Low) (High) (High)
Iteration 2 (Mask & Predict): [Thank] [ ? ] [very] [much] -> [Thank] [ you ] [very] [much]
For models that generate fully in parallel, explicit error-correction layers can be added. Qiu Ran developed RecoverSAT, a framework that accepts parallel outputs and uses a specialized mechanism to detect and recover from multi-modality errors like repetitions and omissions before final output delivery.