TL;DR
Fully non-autoregressive translation (NAT) can generate entire sequences in a single forward pass, but historically suffered significant quality drops due to the assumption that output tokens are conditionally independent. By systematically tackling dependency reduction across four complementary axes—knowledge distillation, continuous latent variables (VAE), latent alignment loss (CTC), and glancing training (GLAT)—this paper produces single-pass NAT models that match autoregressive Transformer baselines while achieving up to a 16.5× inference speedup.
What Problem Does This Paper Address?
Standard autoregressive (AT) machine translation models generate translations token-by-token. While high in quality, sequential decoding suffers from latency bottlenecks. Non-autoregressive translation (NAT) instead predicts all target tokens simultaneously.
However, standard NAT assumes conditional independence among target tokens. Real language data is highly multi-modal and interdependent; ignoring token dependencies causes severe performance degradation (e.g., repeating words or producing inconsistent sentence structures).
Prior work tried to solve this via iterative NAT, which refines translations over several passes. Yet recent benchmarks revealed that simple AT models with deep encoders and shallow 1-layer decoders achieve comparable latency to iterative NAT without sacrificing translation quality. This paper asks: Can a true, single-pass fully NAT model close the translation quality gap without giving up its latency advantage?
Methods
The authors structure their approach around dependency reduction—minimizing target token interdependency across four complementary angles:
Input Sentence
│
▼
[Transformer Encoder] ──► [Continuous Latents (VAE)]
│ │
└───────────► [SoftCopy Up-sampling (3x)]
│
▼
[Transformer Decoder]
│
┌────────────────┴────────────────┐
▼ ▼
[CTC Alignment Loss] [Glancing Targets (GLAT)]
- Data (Knowledge Distillation - KD): NAT models are trained on translations generated by a pre-trained AT teacher model. This simplifies the training distribution, reduces noise, and makes source-target alignments more deterministic.
- Model Architecture (Latent Variables via VAE): The model incorporates continuous latent variables $z \sim \mathcal{N}(\mu, \sigma^2)$ at the encoder output. Trained with a variational lower bound (ELBO), these latent variables capture global sentence-level structure and non-monotonic word reorderings with zero additional decoding iterations.
- Loss Function (Latent Alignments via CTC): Instead of standard cross-entropy that strictly penalizes token shifts (e.g., insertion of a comma), the model uses Connectionist Temporal Classification (CTC). Decoder representations are upsampled (typically 3×), and CTC marginalizes over all valid monotonic alignments using dynamic programming.
- Learning Strategy (Glancing Targets - GLAT): During training, a subset of target tokens is revealed (glanced) to the decoder based on a curriculum proportional to prediction error (Levenshtein distance). The authors adapt GLAT to CTC by glancing target tokens directly along the Viterbi alignment path.
At inference, the model decodes greedily in a single forward pass followed by a fast CTC collapse operation $\Gamma^{-1}$. For hard language pairs, the authors also evaluate CTC beam search (optionally augmented with an $n$-gram language model) and Noisy Parallel Decoding (NPD).
Datasets
The authors evaluated on three standard translation benchmarks:
- WMT14 English $\leftrightarrow$ German (EN $\leftrightarrow$ DE): 4.0M sentence pairs.
- WMT16 English $\leftrightarrow$ Romanian (EN $\leftrightarrow$ RO): 610k sentence pairs.
- WMT20 Japanese $\rightarrow$ English (JA $\rightarrow$ EN): 13M sentence pairs (filtered from 16M using Bicleaner).
No new benchmarks were introduced.
Key Insights and Ideas
- KD and CTC form the essential foundation: Neither KD alone nor CTC alone is sufficient. However, combining them produces a baseline that immediately beats prior state-of-the-art single-pass NAT models across all benchmarks.
- VAEs enable reordering for monotonic CTC: CTC assumes monotonic alignments, which breaks down when translating across languages with drastically different word orders. Qualitative analysis of latent representations revealed that the VAE reorganizes token order in latent space, effectively making the alignment monotonic so CTC can handle it smoothly.
- Iterative NAT is often an unnecessary middle ground: When comparing Pareto frontiers of quality vs. latency (across single-sentence GPU, CPU, and max-batch settings), iterative models fall behind shallow-decoder AT models, whereas single-iteration fully NAT maintains a decisive wall-clock speed advantage.
Key Results
- Matching Autoregressive Quality: On WMT14 EN $\rightarrow$ DE, the proposed NAT model (
KD + CTC + VAE) scored 27.49 BLEU, virtually identical to the standard AT Transformer base teacher (27.48 BLEU) while running 16.5× faster (16.5 ms vs. 272 ms per sentence on a V100 GPU). - Consistent Gains on High/Low-Resource Pairs:
- WMT14 DE $\rightarrow$ EN: 31.39 BLEU (vs. 31.39 AT base).
- WMT16 EN $\rightarrow$ RO: 33.79 BLEU (vs. 33.70 AT base).
- WMT16 RO $\rightarrow$ EN: 34.16 BLEU (vs. 34.05 AT base).
- Distant Language Pairs (JA $\rightarrow$ EN): On the noisy, word-order-divergent WMT20 JA $\rightarrow$ EN benchmark, greedy NAT reached 18.73 BLEU. Adding CTC beam search (beam 20) with a 4-gram LM reached 21.41 BLEU, outperforming the AT Big teacher (21.07 BLEU) while retaining an 11.0× GPU speedup and running over 2× faster on CPU than a lightweight AT base (12-1) model.
Limitations and Future Work
- Memory and Up-sampling Bottleneck: CTC requires upsampling decoder inputs (3× target length), consuming significantly more GPU memory during training and batched inference. The maximum batch size was limited to 15K tokens (compared to 120K tokens for shallow-decoder AT models).
- Distillation Dependency: Knowledge distillation remains practically mandatory; training purely on raw targets causes steep quality drops. Because KD can erase low-frequency words, exploring alternatives like GAN discriminators directly on raw text is a suggested next step.
- Divergent Syntax: Monotonic CTC struggles on linguistically distant pairs without resorting to beam search and language models; exploring richer density estimators (such as normalizing flows) could help bridge this gap natively.
Summary updated