Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation

Yui Oka, Katsuhito Sudoh, Satoshi Nakamura

Abstract

Non-autoregressive neural machine translation (NAT) usually employs sequence-level knowledge distillation using autoregressive neural machine translation (AT) as its teacher model. However, a NAT model often outputs shorter sentences than an AT model. In this work, we propose sequence-level knowledge distillation (SKD) using perturbed length-aware positional encoding and apply it to a student model, the Levenshtein Transformer. Our method outperformed a standard Levenshtein Transformer by 2.5 points in bilingual evaluation understudy (BLEU) at maximum in a WMT14 German to English translation. The NAT model output longer sentences than the baseline NAT models.
BibTeX
@misc{oka2021using,
  author = {Yui Oka and Katsuhito Sudoh and Satoshi Nakamura},
  title = {{Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation}},
  year = {2021},
  eprint = {2107.13689},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.13689},
}

Loading related content...

Figure