Abstract
Non-autoregressive translation (NAT) models significantly improve inference speed compared to autoregressive (AT) models but often suffer from "under-translation," producing outputs shorter than the reference. This paper introduces perturbed length-aware positional encoding (perLDPE), a technique that incorporates target length information into the model's positional encodings during training. By applying perLDPE to both the teacher model during knowledge distillation and the student NAT model, the researchers achieve improvements of up to 2.5 BLEU points on standard benchmarks, effectively mitigating the short-output problem.
Introduction
While modern autoregressive (AT) translation models achieve high accuracy by generating text sequentially, their inference speed is limited. Non-autoregressive (NAT) models address this by generating all tokens in parallel. However, NAT models frequently exhibit a "short output" bias, dropping critical phrases or verbs because they lack the step-by-step feedback inherent in AT generation. Even when trained via knowledge distillation from an AT teacher, NAT students often inherit or amplify this tendency toward brevity.
Methodology
The proposed solution, perturbed length-aware positional encoding (perLDPE), focuses on enhancing the model's awareness of sentence structure relative to its total length.
Length-aware Positional Encoding (LDPE)
Standard Transformer models use sinusoidal positional encodings to denote absolute token positions. LDPE modifies this by incorporating the total sentence length, providing the model with a global "map" of the sequence.
Perturbation and Robustness
To prevent the model from becoming overly dependent on exact length predictions, the authors introduce perturbation. During training, the length value is randomly jittered (e.g., representing a 10-word sentence as 12 words). This noise acts as a regularizer, forcing the model to use length as a flexible guide rather than a rigid constraint.
Dual Application in Distillation
The technique is applied at two stages:
- Teacher Model: Enhancing the AT teacher with perLDPE ensures the distilled data used for training the student is of higher quality and correct length.
- Student Model: In the Levenshtein Transformer (LevT) architecture, perLDPE is applied specifically to the placeholder decoder, which determines where new words should be inserted.
Experimental Setup
The method was evaluated on three benchmarks:
- ASPEC: English-to-Japanese (scientific excerpts).
- WMT14: English-to-German (En-De) and German-to-English (De-En). Tokenization was performed using SentencePiece with vocabularies between 16,000 and 30,000 entries.
Results and Discussion
The experimental results demonstrate that perLDPE significantly improves both translation quality and length fidelity.
Quantitative Performance
On the WMT14 De-En task, the proposed method improved the baseline Levenshtein Transformer by 2.5 BLEU points. Furthermore, the "Length Ratio" (LR)—the ratio of translation length to reference length—showed marked improvement, indicating that the models are less prone to dropping words.
Key Insights
- Teacher Bias Mitigation: Improving the teacher's length awareness prevents the student from learning a "short-output" bias during distillation.
- Oracle Potential: Tests using "Oracle" lengths (exact reference lengths) showed even higher gains (e.g., jumping to 32.1 BLEU on some tasks), suggesting that the translation engine's performance is heavily gated by the accuracy of initial length predictions.
Conclusion
Perturbed length-aware positional encoding effectively addresses the under-translation problem in NAT models. While perLDPE improves how models utilize length information, the initial length prediction remains a bottleneck. Future research may explore more advanced length predictors, such as those based on BERT, to further unlock the potential of this approach.
Summary updated