Parallel Sequence Generation via Non-Autoregressive Transformers

This direction is merged into another direction: non-autoregressive performance gap. Its last version stays readable and citable here.

Version 1 of 134 papers37 conceptsfrozen 2026-10-04

Standard neural machine translation (NMT) models are inherently sequential. To generate a sentence, an autoregressive (AR) model predicts the first word, then uses that word to predict the second, and so on. While this "word-by-word" approach captures the natural flow of language, it creates a massive computational bottleneck: the time it takes to translate grows linearly with the length of the sentence.

Non-Autoregressive Translation (NAT) aims to break this bottleneck by generating all tokens in a sequence simultaneously. By removing the sequential dependency, NAT models can achieve inference speedups of 10x to 20x. However, this speed comes at a cost. Without knowing what its "neighbors" are doing, a NAT model often produces repetitive words or incoherent structures—a challenge known as the multi-modality problem.

The Multi-Modality Problem

The core difficulty of NAT is that a single source sentence can often be translated into several equally valid target sentences. For example, the German "Ich habe Hunger" could be "I am hungry" or "I'm hungry."

An autoregressive model handles this easily: once it picks "I am," the next word is constrained to be "hungry." But a NAT model, predicting all positions at once, might predict "I" at position one, "am" at position two, and "hungry" at position three—or it might accidentally mix two valid translations, resulting in "I'm am hungry." Because tokens are predicted independently, the model lacks a mechanism to ensure they are consistent with one another.

Bridging the Gap with Structural Guidance

To fix this lack of coordination, researchers have introduced "blueprints" or "skeletons" to guide the parallel generation process.

Qiu Ran and colleagues introduced ReorderNAT in Guiding Non-Autoregressive Neural Machine Translation Decoding with Reordering Information. Instead of jumping straight to the target language, the model first predicts how source words should be rearranged. This "pseudo-translation" provides a structural map that helps the model avoid the common NAT pitfall of stuttering or skipping words. Similarly, Jianguo Zhang proposed SNAT in Enriching Non-Autoregressive Transformer with Syntactic and Semantic Structures for Neural Machine Translation, which explicitly teaches the model to recognize parts-of-speech and named entities to ensure the parallel output follows grammatical rules.

Hybrid and Iterative Refinement

Another strategy is to find a middle ground between fully parallel and fully sequential generation.

In Partially Non-Autoregressive Image Captioning, Zhengcong Fei suggests generating groups of words in parallel while maintaining sequential logic between those groups. This "semi-parallel" approach captures local dependencies without the full latency of word-by-word generation.

Taking this further, Xinhui Hu and team developed Hybrid-Regressive Translation (HRT) in Hybrid-Regressive Neural Machine Translation. HRT generates a "skeleton" of the sentence (e.g., every 4th word) autoregressively and then fills in the gaps in a single parallel shot. This ensures the global structure is sound while still being significantly faster than a standard Transformer.

Learning from Autoregressive "Teachers"

If NAT models struggle because they lack the rich internal representations of AR models, why not let them learn from each other?

Shilin He and Zhaopeng Tu explored this in Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine Translation, where a single encoder is shared between an AR and a NAT model. This forces the NAT model to "inherit" the superior linguistic understanding of the AR model.

Xinyou Wang and Shujian Huang took a clever "weak-to-strong" approach in Helping the Weak Makes You Strong: Simple Multi-Task Learning Improves Non-Autoregressive Translators. They trained "weak" AR decoders that were forced to rely on the NAT model's internal features. This pressure forces the NAT model to develop much more "translation-aware" representations, boosting its accuracy without adding any extra time to the final inference process.

Latent Spaces and Diffusion

Recent work has moved toward modeling the "uncertainty" of translation more explicitly.

In Non-Autoregressive Translation by Learning Target Categorical Codes, Shujian Huang introduced CNAT, which uses unsupervised "fuzzy" categorical codes to capture word dependencies. By modeling these codes with a Conditional Random Field (CRF), the model can coordinate its parallel predictions more effectively.

The latest frontier involves Diffusion models, which have revolutionized image generation. Kun Zhou and colleagues proposed Diffusion-NAT in Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation. Instead of generating the sentence in one shot, the model starts with a "noisy" version of the sentence and iteratively refines it. This allows the model to gradually resolve the multi-modality problem, often matching or exceeding the quality of traditional sequential models while remaining highly efficient.

Scaling to Industrial Quality

For a long time, NAT was seen as a research curiosity that couldn't handle real-world complexity. However, Longyue Wang and Zhaopeng Tu demonstrated in Revisiting Non-Autoregressive Translation at Scale that when NAT models are scaled to industrial sizes and paired with specialized architectures like the "Cone" decoder, they can finally close the performance gap with autoregressive models.

The field is moving toward a future where the trade-off between speed and quality is no longer a zero-sum game, but a manageable engineering choice.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parallel_sequence_generation_via_non_aut_baf2d76f,
  title = {Parallel Sequence Generation via Non-Autoregressive Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallel-sequence-generation-via-non-autoregressive-transformers/rdr_baf2d76ffd774ba2b8649823a00158b3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_parallel_sequence_generation_via_non_aut_baf2d76f_v1,
  title = {Parallel Sequence Generation via Non-Autoregressive Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallel-sequence-generation-via-non-autoregressive-transformers/rdr_baf2d76ffd774ba2b8649823a00158b3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}