Standard autoregressive sequence models generate text like a human scribe: one token at a time, left to right, conditioning each new word on every word that came before it. While highly accurate, this sequential bottleneck ($P(y_t \mid y_{<t}, x)$) limits inference speed. Non-autoregressive translation (NAT) and parallel generation models break this bottleneck by predicting all output tokens simultaneously ($P(Y \mid x) = \prod P(y_t \mid x)$).
However, generating an entire sequence in a single parallel step introduces a fundamental challenge known as the multi-modality problem. To overcome this, parallel models rely heavily on a technique called sequence-level knowledge distillation (SeqKD). Rather than acting simply as a method for compressing model size, distillation in this context serves as a critical data-simplification mechanism that fundamentally alters the training distribution.
The Multi-Modality Problem
To understand why parallel generation fails on raw data, consider a source sentence with multiple valid translations. For example, the French phrase "Je vous en prie" can be translated into English as either "You are welcome" or "Don't mention it".
An autoregressive model handles this easily through step-by-step conditioning. If it chooses "You" as the first token, its subsequent probabilities shift to favor "are welcome".
A fully non-autoregressive model, however, emits all tokens independently and simultaneously. Without a mechanism to coordinate choices across positions, the model may predict "You" at the first position (from the first mode) and "mention" at the second position (from the second mode), resulting in an incoherent, garbled mixture like "You mention welcome". As first formalized by Jiatao Gu et al. in Non-Autoregressive Neural Machine Translation, this independent token assumption causes parallel decoders to struggle with the complex, multi-modal nature of human-generated text.
Distillation as a Complexity Filter
The standard solution to the multi-modality problem is to train the parallel "student" model on synthetic targets generated by a high-capacity autoregressive "teacher" model, rather than on raw human translations.
[Source Text] ---> [Autoregressive Teacher] ---> [Deterministic Target Mode] ---> [Parallel Student]
This process works because it acts as a complexity filter. As demonstrated by Chunting Zhou et al. in Understanding Knowledge Distillation in Non-autoregressive Machine Translation, sequence-level knowledge distillation collapses the multimodal distribution of natural language into a single, highly consistent mode. When a deterministic teacher model decodes a dataset (often using beam search), it consistently selects a preferred syntactic structure and vocabulary style for similar inputs.
By training on this distilled data, the student model encounters a target space where the uncertainty of one-to-many mappings is minimized. Yi Ren et al. quantify this effect in A Study of Non-autoregressive Model for Sequence Generation, showing that distillation drastically reduces the "target-token dependency" within the dataset. By stripping away alternative valid paths, the teacher provides a training signal where tokens are much more independent of each other given the source, making the data highly compatible with the parallel model's architectural assumptions.
Quantifying the Simplification
The structural changes that occur when a dataset undergoes sequence-level distillation can be categorized into three main effects:
- Reduction of Conditional Total Correlation: In On the Learning of Non-Autoregressive Transformers, Fei Huang et al. prove that successful parallel training relies on minimizing the conditional total correlation of the target data. The teacher model provides targets that represent a consensus or regularized average of valid references, lowering the statistical entropy of the sequence space.
- Increased Word-Order Monotonicity: Human translations feature rich, non-monotonic reordering patterns (e.g., changing word order dramatically between languages). Weijia Xu et al. show in How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation? that distillation simplifies word order and reduces vocabulary variety. The resulting synthetic text exhibits a more monotonic, word-for-word alignment with the source text.
- Predictable Sequence Lengths: Because parallel models must allocate a fixed number of token slots before generation begins, predicting the target length is a crucial first step. Minghan Wang et al. demonstrate in How Length Prediction Influence the Performance of Non-Autoregressive Translation? that the structural regularity of distilled text makes target sequence lengths significantly more predictable, directly boosting generation quality.
Without this pre-processing step to reduce target diversity, the performance of fully parallel models severely degrades. As noted by Robin M. Schmidt in Non-Autoregressive Neural Machine Translation: A Call for Clarity, this heavy reliance on pre-processed, simplified data distributions remains a defining characteristic of the field.
Advanced Distillation Paradigms
As the field has matured, researchers have developed more nuanced ways to manipulate the target distribution through distillation:
- Intermediate Checkpoint Distillation: Rather than using a fully converged, highly complex teacher model, the authors of Language Modeling, Lexical Translation, Reordering: The Training Process of NMT through the Lens of Classical SMT discover that distilling from intermediate teacher checkpoints can be more effective. Early-stage teachers naturally produce more monotonic, literal translations, offering an even simpler learning signal for the parallel student.
- Cross-Architectural Alignment: In tasks like generative masked language modeling, independent sampling limitations still cause local errors like token stuttering. Yuchen Li et al. show in Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines that cross-architectural distillation provides a smoother, more consistent target distribution that helps parallel models internalize complex sequence patterns without explicit left-to-right decoding.
- Multi-Reference Selection: While collapsing the target distribution to a single mode makes learning easier, it can also restrict the student's expressive capacity. To balance simplicity and diversity, Chenze Shao et al. introduce a multi-reference distillation framework in One Reference Is Not Enough: Diverse Distillation with Reference Selection for Non-Autoregressive Translation. This approach provides multiple high-quality teacher options and allows the parallel model to dynamically select the target that best aligns with its own internal predictions, mitigating multi-modality without completely sacrificing target diversity.
Key Foundational Work
Mask-Predict: Parallel Decoding of Conditional Masked Language Models (2019) — contributed 2 concept elements to this direction.
Non-Autoregressive Neural Machine Translation (2018) — contributed 2 concept elements to this direction.
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018) — contributed 1 concept elements to this direction.