Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation

Jason Lee, Raphael Shu, Kyunghyun Cho

Abstract

We propose an efficient inference procedure for non-autoregressive machine translation that iteratively refines translation purely in the continuous space. Given a continuous latent variable model for machine translation, we train an inference network to approximate the gradient of the marginal log probability of the target sentence, using only the latent variable as input. This allows us to use gradient-based optimization to find the target sentence at inference time that approximately maximizes its marginal probability. As each refinement step only involves computation in the latent space of low dimensionality (we use 8 in our experiments), we avoid computational overhead incurred by existing non-autoregressive inference procedures that often refine in token space. We evaluate our approach on WMT'14 En-De, WMT'16 Ro-En and IWSLT'16 De-En, and observe that it is computationally efficient and more effective than EM-like inference, resulting in higher marginal probabilities and BLEU scores. On WMT'14 En-De, our approach is 6.2 times faster than the autoregressive model with minimal degradation to translation quality (0.9 BLEU).

Introduction

Most modern neural machine translation (NMT) systems are autoregressive, generating text sequentially. While accurate, this linear dependency on sentence length results in slow inference. Non-autoregressive translation (NAT) models aim to generate all tokens simultaneously to improve speed. However, NAT models often suffer from "multimodal" errors, where the model produces inconsistent or repetitive outputs due to the lack of sequential dependencies.

To mitigate this, iterative refinement is often employed, where a draft translation is polished over several steps. Traditional refinement occurs in the discrete "token space," requiring the model to evaluate the entire vocabulary (often 30,000+ tokens) at each step. This process is computationally expensive and significantly reduces the speed advantages of NAT.

Proposed Method: Continuous Latent Refinement

This paper proposes performing iterative refinement entirely within a low-dimensional continuous latent space. The approach utilizes a latent variable model where a sentence is represented by a continuous variable $z$. A decoder then maps $z$ to the final target sentence.

The core innovation is the training of an auxiliary inference network to estimate the gradient of the translation's marginal log probability. Two parameterization methods were explored:

  1. Energy Function: A network predicts a quality score for a point in latent space, requiring backpropagation to find the gradient.
  2. Score Function: A network directly predicts the gradient vector, offering superior speed by bypassing backpropagation during inference.

At test time, the model starts with an initial guess in the latent space and iteratively "walks" in the direction suggested by the inference network before decoding the final output.

Experimental Results

The approach was evaluated on WMT’14 En→De, WMT’16 Ro→En, and IWSLT’16 De→En benchmarks using knowledge distillation.

Efficiency and Effectiveness

By refining in an 8-dimensional latent space, the model avoids the overhead of discrete token-space search. The "Score" function approach proved particularly effective, being twice as fast as previous hybrid methods like "Delta Inference."

Figure 1: Comparison of marginal log-likelihood (LL) across refinement steps for Delta, Score, and Energy methods.

As shown in Figure 1, the Score-based refinement effectively increases the marginal log-likelihood over successive steps, outperforming the Energy-based baseline and matching or exceeding the hybrid Delta method.

Qualitative Improvements

One significant advantage of continuous refinement is the reduction of common NAT errors, such as word repetition.

Figure 2: Edit distance and repetition rate across refinement steps.

Figure 2 demonstrates that the Score-based method leads to a higher edit distance from the initial draft (indicating more substantial refinement) and a significantly lower repetition rate compared to the Delta method. On the WMT’14 En→De task, the Score model achieved a 6.2x speedup over a standard Transformer with only a 0.9 BLEU point reduction in quality.

Conclusion

Refining translations in a continuous latent space offers a compelling balance between speed and quality for NAT. While a small quality gap remains compared to autoregressive models, the significant speedups and reduction in repetitive errors make this a promising direction. Future work may explore more complex prior distributions and direct training on the true marginal likelihood gradient to further close the performance gap.

Summary updated

Abstract

We propose an efficient inference procedure for non-autoregressive machine translation that iteratively refines translation purely in the continuous space. Given a continuous latent variable model for machine translation (Shu et al., 2020), we train an inference network to approximate the gradient of the marginal log probability of the target sentence, using only the latent variable as input. This allows us to use gradient-based optimization to find the target sentence at inference time that approximately maximizes its marginal probability. As each refinement step only involves computation in the latent space of low dimensionality (we use 8 in our experiments), we avoid computational overhead incurred by existing non-autoregressive inference procedures that often refine in token space. We compare our approach to a recently proposed EM-like inference procedure (Shu et al., 2020) that optimizes in a hybrid space, consisting of both discrete and continuous variables. We evaluate our approach on WMT'14 En-De, WMT'16 Ro-En and IWSLT'16 De-En, and observe two advantages over the EM-like inference: (1) it is computationally efficient, i.e. each refinement step is twice as fast, and (2) it is more effective, resulting in higher marginal probabilities and BLEU scores with the same number of refinement steps. On WMT'14 En-De, for instance, our approach is able to decode 6.2 times faster than the autoregressive model with minimal degradation to translation quality (0.9 BLEU).
BibTeX
@inproceedings{lee2020iterative,
  author = {Jason Lee and Raphael Shu and Kyunghyun Cho},
  title = {{Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=HpfEAT4H6kq},
}
Figure