Hybrid-Regressive Neural Machine Translation

Qiang Wang, Xinhui Hu, Ming ChenarXiv 2022 · DOI · arXiv

Abstract

In this work, we empirically confirm that non-autoregressive translation with an iterative refinement mechanism (IR-NAT) suffers from poor acceleration robustness because it is more sensitive to decoding batch size and computing device setting than autoregressive translation (AT). Inspired by it, we attempt to investigate how to combine the strengths of autoregressive and non-autoregressive translation paradigms better. To this end, we demonstrate through synthetic experiments that prompting a small number of AT's predictions can promote one-shot non-autoregressive translation to achieve the equivalent performance of IR-NAT. Following this line, we propose a new two-stage translation prototype called hybrid-regressive translation (HRT). Specifically, HRT first generates discontinuous sequences via autoregression (e.g., make a prediction every k tokens, k>1) and then fills in all previously skipped tokens at once in a non-autoregressive manner. We also propose a bag of techniques to effectively and efficiently train HRT without adding any model parameters. HRT achieves the state-of-the-art BLEU score of 28.49 on the WMT En-De task and is at least 1.5x faster than AT, regardless of batch size and device. In addition, another bonus of HRT is that it successfully inherits the good characteristics of AT in the deep-encoder-shallow-decoder architecture. Concretely, compared to the vanilla HRT with a 6-layer encoder and 6-layer decoder, the inference speed of HRT with a 12-layer encoder and 1-layer decoder is further doubled on both GPU and CPU without BLEU loss.

Introduction

Neural Machine Translation (NMT) has long been dominated by the Autoregressive Translation (AT) paradigm, which generates tokens sequentially. While AT ensures high translation quality by modeling strong dependencies between tokens, its sequential nature leads to high latency. Non-Autoregressive Translation (NAT) models were introduced to mitigate this by predicting all tokens in parallel. However, vanilla NAT often suffers from the "multimodality problem," leading to lower quality.

To address this, researchers developed Iterative Refinement NAT (IR-NAT), which polishes translations over multiple passes. Despite their promise, this paper identifies a critical weakness in IR-NAT: Acceleration Robustness.

Relative speedup ratio vs. decoding batch size for iterative NAT models.

As illustrated in Figure 1, while IR-NAT models like CMLM show significant speedups for small batch sizes, their relative advantage over AT drops sharply as batch size increases or when running on CPUs. This lack of robustness limits their practical utility in high-throughput production environments.

Methodology: Hybrid-Regressive Translation (HRT)

The authors propose Hybrid-Regressive Translation (HRT), a two-stage approach designed to combine the global coherence of AT with the speed of NAT. The core idea is inspired by the "Prompting Power" observed in synthetic experiments: providing a few correctly placed tokens allows a one-shot NAT model to match the performance of complex iterative models.

Two-Stage Decoding

  1. Stage I: Skip-AT (Skeleton Generation): An autoregressive decoder generates a discontinuous sequence by skipping tokens at a fixed interval $k$. For example, with $k=2$, the model predicts every second word. This provides a globally coherent "skeleton" for the sentence.
  2. Stage II: Skip-CMLM (Gap Filling): A non-autoregressive decoder takes this skeleton as a prompt and fills in all missing gaps simultaneously in a single parallel step.
Architecture of the TASK-SKIP-AT stage.

Training Strategies

To ensure effective learning without increasing model parameters, HRT employs several techniques:

  • Multi-Task Framework: The model is trained on four tasks: Skip-AT, Skip-CMLM, standard AT, and standard Masked Language Modeling (MLM). This helps the model learn both local token dependencies and global sentence structure.
  • Curriculum Learning: Training begins with a focus on auxiliary tasks (standard AT/MLM) and gradually shifts to the more challenging "skip" tasks.
  • Mixed Distillation: The training set is a mixture of raw data and distilled data (outputs from a teacher model), preventing the model from over-simplifying the target language.

Experimental Results

HRT was evaluated on WMT’16 English-Romanian (En-Ro), WMT’14 English-German (En-De), and NIST Chinese-English (Zh-En).

Performance and Speed

HRT achieved a state-of-the-art BLEU score of 28.49 on WMT En-De, outperforming most existing NAT and semi-NAT models. Unlike IR-NAT, HRT maintains a consistent speedup of at least 1.5x over standard Transformers across all tested batch sizes and computing devices (GPU and CPU).

Efficiency Gains with Shallow Decoders

A key advantage of HRT is its ability to inherit AT's performance in "deep-encoder-shallow-decoder" architectures. By utilizing a 12-layer encoder and a 1-layer decoder (HRT-20L), the model achieves up to 4x acceleration on GPU and 3x on CPU compared to the baseline, without any loss in translation quality.

Conclusion

Hybrid-Regressive Translation (HRT) offers a robust solution to the speed-quality trade-off in NMT. By leveraging a two-stage "skeleton-and-fill" approach, it achieves high translation quality while maintaining significant and robust acceleration across diverse hardware environments. Future research will focus on optimizing the skip interval $k$ and exploring more adaptive skeleton generation methods.

Summary updated

Abstract

In this work, we empirically confirm that non-autoregressive translation with an iterative refinement mechanism (IR-NAT) suffers from poor acceleration robustness because it is more sensitive to decoding batch size and computing device setting than autoregressive translation (AT). Inspired by it, we attempt to investigate how to combine the strengths of autoregressive and non-autoregressive translation paradigms better. To this end, we demonstrate through synthetic experiments that prompting a small number of AT's predictions can promote one-shot non-autoregressive translation to achieve the equivalent performance of IR-NAT. Following this line, we propose a new two-stage translation prototype called hybrid-regressive translation (HRT). Specifically, HRT first generates discontinuous sequences via autoregression (e.g., make a prediction every k tokens, k>1) and then fills in all previously skipped tokens at once in a non-autoregressive manner. We also propose a bag of techniques to effectively and efficiently train HRT without adding any model parameters. HRT achieves the state-of-the-art BLEU score of 28.49 on the WMT En-De task and is at least 1.5x faster than AT, regardless of batch size and device. In addition, another bonus of HRT is that it successfully inherits the good characteristics of AT in the deep-encoder-shallow-decoder architecture. Concretely, compared to the vanilla HRT with a 6-layer encoder and 6-layer decoder, the inference speed of HRT with a 12-layer encoder and 1-layer decoder is further doubled on both GPU and CPU without BLEU loss.
BibTeX
@misc{wang2022hybrid,
  author = {Qiang Wang and Xinhui Hu and Ming Chen},
  title = {{Hybrid-Regressive Neural Machine Translation}},
  year = {2022},
  eprint = {2210.10416},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.10416},
}
Figure