Abstract: Jason Lee, Raphael Shu, Kyunghyun Cho. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Standard Neural Machine Translation (NMT) models typically operate autoregressively, generating text one token at a time from left to right. While this sequential approach ensures high translation quality by capturing local dependencies, it introduces a significant computational bottleneck: decoding latency scales linearly with sentence length. Non-Autoregressive Translation (NAT) models aim to solve this by generating all tokens in parallel. However, NAT models often suffer from the "multimodality problem," where the model struggles to coordinate dependencies between tokens without a step-by-step history, leading to errors like repetitions or missing words.
To bridge the quality gap, many NAT models employ iterative refinement, where an initial draft is repeatedly polished. Traditional refinement occurs in the discrete token space—searching through a vocabulary of tens of thousands of entries—which is computationally expensive. This paper proposes a novel alternative: performing iterative refinement in a low-dimensional continuous latent space.
Continuous Latent Space Refinement
The authors build upon a latent variable model where a continuous variable $z$ captures the global structure and semantic intent of a sentence. Instead of editing words directly, the model refines the latent representation $z$ before decoding it into a final sentence. The core of this approach is a new inference procedure involving a "score function."
Learning a Score Function
The model is trained to estimate the gradient of the marginal log-probability of a target sentence with respect to the latent variable $z$. Because calculating the true gradient through a discrete decoding process is difficult, the authors utilize "proxy gradients" derived from Delta inference (an EM-like method). The network learns to predict how $z$ should be adjusted to maximize translation likelihood.
As shown in Figure 1, the proposed "Score" method effectively increases the log-likelihood (LL) of the translation over successive steps, performing comparably to the more complex Delta inference while maintaining higher efficiency. By refining in a latent space of only 8 dimensions, the model can make "non-local" changes—such as reordering entire phrases—with a single gradient step, a feat much harder to achieve in discrete space.
Experimental Results
The proposed method was evaluated on standard benchmarks, including WMT’14 En→De, WMT’16 Ro→En, and IWSLT’16 De→En. To optimize performance, the NAT models were trained using knowledge distillation from a high-quality autoregressive Transformer.
Speed and Efficiency
The "Score" function approach achieved a 6.2x speedup over the autoregressive Transformer baseline on WMT’14 En→De. Crucially, each refinement step in the continuous space was approximately twice as fast as previous state-of-the-art hybrid methods that combined discrete and continuous refinement. This efficiency stems from bypassing the "vocabulary bottleneck," where models must compute a softmax over 30k+ words at every iteration.
Translation Quality
The method significantly closed the performance gap with autoregressive models. On En→De, the model reached within 0.9 BLEU points of the baseline. On IWSLT’16 De→En, four steps of refinement improved the BLEU score by 2.1 points compared to the initial parallel generation (0 steps).
One of the most notable improvements is the reduction of token repetitions, a hallmark of the multimodality problem in NAT. As illustrated in Figure 3, the Score-based refinement drastically reduces the repetition rate compared to the initial generation, outperforming the Delta inference baseline in producing cleaner, more coherent text.
Conclusion and Future Work
By shifting the refinement process from the discrete vocabulary space to a low-dimensional continuous latent space, this research demonstrates that NAT models can achieve high translation quality without sacrificing their inherent speed advantages. While the current model uses a simple spherical Gaussian prior and fixed length prediction, future work may explore more complex priors like Normalizing Flows to further enhance the representational power of the latent space. Despite these remaining challenges, iterative refinement in continuous space represents a major step toward making non-autoregressive translation a practical reality for real-time applications.
Summary updated