Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade

Jiatao Gu, Xiang Kong

Abstract: Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with a single forward pass of neural networks, which significantly reduces inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves new state-of-the-art results for fully NAT models and obtains comparable performance with autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speedup at inference time.

The Challenge of Non-autoregressive Translation

Standard Neural Machine Translation (NMT) relies on autoregressive (AT) models, like the Transformer, which generate text sequentially—one word at a time. Because each step depends on the previous one, inference latency is high, especially for long sentences.

Non-autoregressive translation (NAT) models attempt to solve this by generating all tokens in parallel. However, vanilla NAT models assume that output tokens are conditionally independent of each other. In real language, this is false (e.g., the word "Vielen" strongly predicts the word "Dank" in German). Failing to capture these dependencies causes NAT models to output repetitive or incoherent text, leading to a massive drop in translation quality.

To fix this, recent state-of-the-art NAT models use an iterative refinement process, taking multiple passes to fix errors. However, iterative NAT models often lose their speed advantage when compared to properly optimized AT models. The core challenge this paper tackles is: How can we build a fully NAT model (a single parallel forward pass) that matches the quality of autoregressive models without sacrificing its massive speed advantage?

Methods: Dependency Reduction via Four "Tricks"

The authors propose a guiding principle for designing NAT models: dependency reduction. The goal is to make the target tokens as independent as possible in the learning space so the NAT model can easily predict them in parallel. They achieve this by combining four "tricks of the trade" into a unified architecture.

The proposed NAT architecture integrating latent variables, CTC loss, and glancing targets.
  1. Data (Knowledge Distillation): Instead of training on raw human translations, the NAT model is trained on translations generated by a strong AT "teacher" model. This simplifies the training data, removing noise and making the mappings more deterministic.
  2. Model (Latent Variables / VAE): The authors introduce continuous latent variables at the encoder output. This allows the model to capture complex, non-deterministic dependencies before the parallel decoding step.
  3. Loss Function (Latent Alignments / CTC): Standard Cross-Entropy loss heavily penalizes a model if it outputs the correct translation but shifted by one position. The authors use Connectionist Temporal Classification (CTC) loss, which marginalizes over all valid alignments, allowing the model to focus on token accuracy rather than rigid positioning.
  4. Learning Strategy (Glancing Targets / GLAT): They employ a curriculum learning strategy where the model is allowed to "glance" at some of the correct target tokens during training. As the model's predictions improve, the number of masked words increases, gradually teaching it to generate the whole sentence independently.

Key Insights and Results

The models were evaluated on three standard benchmarks: WMT14 English-German (EN-DE), WMT16 English-Romanian (EN-RO), and WMT20 Japanese-English (JA-EN).

Comparison of BLEU scores and inference speed-up across various NAT models and the Transformer baseline.
  • Closing the Gap: On WMT14 EN-DE, the fully NAT model achieved 27.49 BLEU, practically identical to the AT Transformer baseline (27.48 BLEU), while providing a 16.5x speedup during inference.
  • Beating the Teacher: On the difficult WMT20 JA-EN dataset, the fully NAT model (using CTC beam search) achieved 21.84 BLEU, actually outperforming the AT teacher model (21.07 BLEU) while remaining 6.0x faster.
  • Synergy is Required: Ablation studies showed that no single trick solves the NAT independence problem. Knowledge Distillation and CTC loss are foundational, while Latent Variables and Glancing Targets provide the final push to capture non-monotonic dependencies like word reordering.

Limitations and Future Work

Despite these gains, challenges remain. To use CTC loss, the encoder output must be upsampled, which increases memory consumption and can limit batch sizes in production environments. Additionally, while Knowledge Distillation is crucial, it can cause the model to lose information about low-frequency words. Future work may explore using GANs to train NAT models directly on raw targets or employing normalizing flows to capture even richer dependency information for linguistically distant language pairs.

Summary updated

Abstract

Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks, which significantly reduces the inference latency at the expense of quality drop compared to the Transformer baseline. In this work, we target on closing the performance gap while maintaining the latency advantage. We first inspect the fundamental issues of fully NAT models, and adopt dependency reduction in the learning space of output tokens as the basic guidance. Then, we revisit methods in four different aspects that have been proven effective for improving NAT models, and carefully combine these techniques with necessary modifications. Our extensive experiments on three translation benchmarks show that the proposed system achieves the new state-of-the-art results for fully NAT models, and obtains comparable performance with the autoregressive and iterative NAT systems. For instance, one of the proposed models achieves 27.49 BLEU points on WMT14 En-De with approximately 16.5X speed up at inference time.
BibTeX
@inproceedings{gu2021fully,
  author = {Jiatao Gu and Xiang Kong},
  title = {{Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=_bNWLuQdD19},
}
Figure