Parallel Sequence Generation with Non Autoregressive Transformers

This direction is merged into another direction: non-autoregressive performance gap. Its last version stays readable and citable here.

Version 1 of 129 papers30 conceptsfrozen 2026-10-04

Standard sequence-to-sequence models, such as the original Transformer, are autoregressive: they generate text one token at a time. Each new word depends on all previously generated words, creating a sequential bottleneck where the time complexity scales linearly with the length of the output.

Non-Autoregressive Translation (NAT) aims to break this bottleneck by generating all tokens in a sequence simultaneously. While this offers massive speedups—often 15x to 30x faster than traditional models—it introduces a fundamental challenge: how do you maintain coherence when tokens cannot "see" what their neighbors are choosing?

The Multimodality Problem

The primary obstacle in parallel generation is the multimodality problem. In an autoregressive model, if there are two valid ways to translate a sentence, the first token choice constrains all subsequent choices. In a non-autoregressive model, different parts of the decoder might independently pick tokens from different valid translations.

For example, if a model is translating a phrase that could be "thank you" or "merci," a parallel decoder might produce "thank merci" because the first position chose the English path while the second chose the French path. This lack of coordination leads to the "performance deficit" noted by Jiatao Gu and colleagues in their foundational work, Non-Autoregressive Neural Machine Translation.

Planning with Latent Variables and Fertility

To solve this coordination problem, researchers introduced mechanisms to "plan" the sentence structure before generating tokens. Jiatao Gu et al. proposed the concept of fertility, a latent variable that predicts how many output tokens each input token should produce. By deciding the length and layout of the sentence upfront, the model provides a shared blueprint that helps parallel heads stay in sync.

In the domain of speech, Wei Ping and the creators of ClariNet used Gaussian Inverse Autoregressive Flows to capture complex temporal dependencies in audio waveforms, allowing for high-fidelity speech synthesis at speeds 20x faster than real-time.

Bridging the Gap with Knowledge Distillation

Since autoregressive (AR) models are naturally better at capturing linguistic nuances, many NAT approaches use them as "teachers." Di He and Zhuohan Li introduced Hint-Based Training, where a pre-trained AR model provides "hints" regarding hidden states and word alignments. This guidance helps the NAT model learn the internal logic of sequential dependencies without needing to execute them at inference time.

Similarly, Mahmoud G. Salem and Frederick Liu demonstrated in UT5 that large-scale pretraining with "unrolled denoising" can significantly close the quality gap, allowing non-autoregressive models to perform competitively on complex tasks like summarization (XSum) and question answering (SQuAD).

Flexible Alignment and Loss Functions

Traditional training uses Cross-Entropy loss, which penalizes a model if it predicts the right word but in the wrong position. For NAT models, which often suffer from minor positional shifts, this penalty is too harsh.

To address this, Marjan Ghazvininejad and Luke Zettlemoyer developed Aligned Cross Entropy (AXE). AXE uses dynamic programming to find the best monotonic alignment between the model's parallel predictions and the target sentence. By allowing for slight offsets, the loss function focuses on the content of the translation rather than strict token-to-position matching, drastically reducing word repetitions.

Iterative Refinement and Diffusion

The most recent wave of research moves away from "pure" one-pass generation toward high-speed iterative refinement.

  • RenewNAT: Pei Guo and Juntao Li 0005 proposed RenewNAT, which uses a dual-module decoder to generate a "potential translation" and refine it in a single pass.
  • Diffusion: Lihua Qian et al. introduced Diff-Glat, which treats sequence generation as a denoising process. By combining discrete modality diffusion with "glancing sampling," the model iteratively cleans a noisy initial signal into a coherent sentence.
  • Aggressive Decoding: Tao Ge and Furu Wei explored a "draft-then-verify" paradigm in Lossless Acceleration for Seq2seq Generation. This approach uses a fast NAT-like drafter to propose multiple tokens and a standard AR model to verify them in parallel, achieving speedups without any loss in quality.

Through these diverse strategies—latent planning, teacher guidance, flexible alignment, and iterative refinement—the field is successfully decoupling generation quality from sequential execution, bringing us closer to instantaneous, high-fidelity machine translation.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parallel_sequence_generation_with_non_au_4b158eb2,
  title = {Parallel Sequence Generation with Non Autoregressive Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallel-sequence-generation-with-non-autoregressive-transformers/rdr_4b158eb291984bfd9aff00f5ad0b6bd3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_parallel_sequence_generation_with_non_au_4b158eb2_v1,
  title = {Parallel Sequence Generation with Non Autoregressive Transformers},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parallel-sequence-generation-with-non-autoregressive-transformers/rdr_4b158eb291984bfd9aff00f5ad0b6bd3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}