Improving Autoregressive NMT with Non-Autoregressive Model

Long Zhou, Jiajun Zhang, Chengqing Zong2020 · DOI

Abstract

Autoregressive neural machine translation (NMT) models are often used to teach nonautoregressive models via knowledge distillation. However, there are few studies on improving the quality of autoregressive translation (AT) using non-autoregressive translation (NAT). In this work, we propose a novel Encoder-NAD-AD framework for NMT, aiming at boosting AT with global information produced by NAT model. Specifically, under the semantic guidance of source-side context captured by the encoder, the nonautoregressive decoder (NAD) first learns to generate target-side hidden state sequence in parallel. Then the autoregressive decoder (AD) performs translation from left to right, conditioned on source-side and target-side hidden states. Since AD has global information generated by low-latency NAD, it is more likely to produce a better translation with less time delay. Experiments on WMT14 EnDe, WMT16 EnRo, and IWSLT14 DeEn translation tasks demonstrate that our framework achieves significant improvements with only 8% speed degeneration over the autoregressive NMT.

Introduction

In Neural Machine Translation (NMT), researchers typically face a trade-off between speed and quality. Autoregressive (AT) models, like the standard Transformer, generate tokens sequentially, ensuring high quality through strong local dependencies but suffering from slow inference. Conversely, Non-Autoregressive (NAT) models generate all tokens simultaneously, offering high speed but often at the cost of global structural consistency.

While AT models are frequently used as "teachers" for NAT models via knowledge distillation, this research explores the inverse: leveraging the speed of NAT to enhance the quality of AT. The proposed Encoder-NAD-AD framework uses a fast NAT component to provide a "global draft" that guides a standard AT decoder, granting it a form of "future sight" to improve translation accuracy with minimal latency overhead.

The Encoder-NAD-AD Framework

The architecture consists of three primary modules designed to integrate global context into the sequential generation process:

  1. Neural Encoder: A standard Transformer encoder that converts the source sentence into a sequence of vector representations.
  2. Non-Autoregressive Decoder (NAD): This module acts as a "fast drafter." It processes the encoder's output to generate a complete sequence of hidden states for the target sentence in parallel. Due to its non-autoregressive nature, this step is nearly instantaneous.
  3. Autoregressive Decoder (AD): The "refiner" module generates tokens one by one. It incorporates a unique NAD-AD Cross-Attention layer, allowing it to attend to the global hidden states produced by the NAD at every step.
The Encoder-NAD-AD framework architecture, showing the parallel generation of the NAD draft followed by the sequential AD refinement.

By combining these elements, the AD is no longer limited to its own previous outputs; it can leverage the global gist of the target sentence provided by the NAD to make more informed linguistic choices.

Experimental Evaluation

The framework was evaluated on three standard translation benchmarks:

  • WMT14 English⇒German (En⇒De): 4.5M sentence pairs.
  • WMT16 English⇒Romanian (En⇒Ro): 600K sentence pairs.
  • IWSLT14 German⇒English (De⇒En): 153K sentence pairs.

Key Results and Insights

The proposed model demonstrated superior performance across all tested datasets. As shown in the results table below, it consistently outperformed both standard Transformer baselines and existing NAT/AT systems.

Comparison of BLEU scores across different NMT systems and datasets. "Our Model" shows significant improvements over the Transformer baseline.

Performance Gains

  • Quality: The model achieved BLEU score improvements of +0.59 on En⇒De, +0.89 on En⇒Ro, and +1.14 on De⇒En compared to the Transformer baseline.
  • Efficiency: Unlike previous multi-pass approaches like Deliberate Networks, which can slow down inference by nearly 70%, this framework only incurs an 8% speed degeneration. This efficiency is due to the inherent speed of the NAT component, which provides a 16x speedup over typical AT decoders.
  • Global Context: The NAD does not need to produce a perfect translation; its role is to provide a useful hidden state sequence. This "global vision" helps the AT decoder avoid local optima and structural errors common in purely sequential models.

Conclusion and Future Work

The Encoder-NAD-AD framework successfully bridges the gap between NAT speed and AT quality. While the model increases parameter count and still benefits from knowledge distillation during training, it offers a practical path toward higher-quality NMT with negligible latency costs. Future research may focus on adapting this two-pass approach for incremental, real-time simultaneous translation tasks.

Summary updated

Abstract

Autoregressive neural machine translation (NMT) models are often used to teach nonautoregressive models via knowledge distillation. However, there are few studies on improving the quality of autoregressive translation (AT) using non-autoregressive translation (NAT). In this work, we propose a novel Encoder-NAD-AD framework for NMT, aiming at boosting AT with global information produced by NAT model. Specifically, under the semantic guidance of source-side context captured by the encoder, the nonautoregressive decoder (NAD) first learns to generate target-side hidden state sequence in parallel. Then the autoregressive decoder (AD) performs translation from left to right, conditioned on source-side and target-side hidden states. Since AD has global information generated by low-latency NAD, it is more likely to produce a better translation with less time delay. Experiments on WMT14 EnDe, WMT16 EnRo, and IWSLT14 DeEn translation tasks demonstrate that our framework achieves significant improvements with only 8% speed degeneration over the autoregressive NMT.
BibTeX
@misc{zhou2020improving,
  author = {Long Zhou and Jiajun Zhang and Chengqing Zong},
  title = {{Improving Autoregressive NMT with Non-Autoregressive Model}},
  year = {2020},
  doi = {10.18653/v1/2020.autosimtrans-1.4},
}
Figure