Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation

Yui Oka, Katsuhito Sudoh, Satoshi Nakamura

Abstract

Non-autoregressive translation (NAT) models significantly improve inference speed compared to autoregressive (AT) models but often suffer from "under-translation," producing outputs shorter than the reference. This paper introduces perturbed length-aware positional encoding (perLDPE), a technique that incorporates target length information into the model's positional encodings during training. By applying perLDPE to both the teacher model during knowledge distillation and the student NAT model, the researchers achieve improvements of up to 2.5 BLEU points on standard benchmarks, effectively mitigating the short-output problem.

Introduction

While modern autoregressive (AT) translation models achieve high accuracy by generating text sequentially, their inference speed is limited. Non-autoregressive (NAT) models address this by generating all tokens in parallel. However, NAT models frequently exhibit a "short output" bias, dropping critical phrases or verbs because they lack the step-by-step feedback inherent in AT generation. Even when trained via knowledge distillation from an AT teacher, NAT students often inherit or amplify this tendency toward brevity.

Methodology

The proposed solution, perturbed length-aware positional encoding (perLDPE), focuses on enhancing the model's awareness of sentence structure relative to its total length.

Length-aware Positional Encoding (LDPE)

Standard Transformer models use sinusoidal positional encodings to denote absolute token positions. LDPE modifies this by incorporating the total sentence length, providing the model with a global "map" of the sequence.

Perturbation and Robustness

To prevent the model from becoming overly dependent on exact length predictions, the authors introduce perturbation. During training, the length value is randomly jittered (e.g., representing a 10-word sentence as 12 words). This noise acts as a regularizer, forcing the model to use length as a flexible guide rather than a rigid constraint.

Dual Application in Distillation

The technique is applied at two stages:

  1. Teacher Model: Enhancing the AT teacher with perLDPE ensures the distilled data used for training the student is of higher quality and correct length.
  2. Student Model: In the Levenshtein Transformer (LevT) architecture, perLDPE is applied specifically to the placeholder decoder, which determines where new words should be inserted.
The training and distillation pipeline using perLDPE for both teacher and student models.

Experimental Setup

The method was evaluated on three benchmarks:

  • ASPEC: English-to-Japanese (scientific excerpts).
  • WMT14: English-to-German (En-De) and German-to-English (De-En). Tokenization was performed using SentencePiece with vocabularies between 16,000 and 30,000 entries.

Results and Discussion

The experimental results demonstrate that perLDPE significantly improves both translation quality and length fidelity.

Quantitative Performance

On the WMT14 De-En task, the proposed method improved the baseline Levenshtein Transformer by 2.5 BLEU points. Furthermore, the "Length Ratio" (LR)—the ratio of translation length to reference length—showed marked improvement, indicating that the models are less prone to dropping words.

Main experimental results comparing standard Transformers and NAT models with and without perLDPE across multiple language pairs.

Key Insights

  • Teacher Bias Mitigation: Improving the teacher's length awareness prevents the student from learning a "short-output" bias during distillation.
  • Oracle Potential: Tests using "Oracle" lengths (exact reference lengths) showed even higher gains (e.g., jumping to 32.1 BLEU on some tasks), suggesting that the translation engine's performance is heavily gated by the accuracy of initial length predictions.

Conclusion

Perturbed length-aware positional encoding effectively addresses the under-translation problem in NAT models. While perLDPE improves how models utilize length information, the initial length prediction remains a bottleneck. Future research may explore more advanced length predictors, such as those based on BERT, to further unlock the potential of this approach.

Summary updated

Abstract

Non-autoregressive neural machine translation (NAT) usually employs sequence-level knowledge distillation using autoregressive neural machine translation (AT) as its teacher model. However, a NAT model often outputs shorter sentences than an AT model. In this work, we propose sequence-level knowledge distillation (SKD) using perturbed length-aware positional encoding and apply it to a student model, the Levenshtein Transformer. Our method outperformed a standard Levenshtein Transformer by 2.5 points in bilingual evaluation understudy (BLEU) at maximum in a WMT14 German to English translation. The NAT model output longer sentences than the baseline NAT models.
BibTeX
@misc{oka2021using,
  author = {Yui Oka and Katsuhito Sudoh and Satoshi Nakamura},
  title = {{Using Perturbed Length-aware Positional Encoding for Non-autoregressive Neural Machine Translation}},
  year = {2021},
  eprint = {2107.13689},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.13689},
}

Loading related content...

Figure