TL;DR
This paper introduces a "local autoregressive translation" (LAT) mechanism to improve non-autoregressive translation (NAT) models. By predicting short sequences of tokens autoregressively at each decoding step and then efficiently merging them, LAT captures local dependencies, leading to better translation quality and reduced repetition, all while achieving significant speedups.
What Problem Does This Paper Address?
Traditional neural machine translation (NMT) models are autoregressive, meaning they generate translations token by token, which is accurate but slow. Non-autoregressive translation (NAT) models aim to speed this up by generating multiple tokens simultaneously. However, NAT models often struggle with the "conditional independence" assumption, leading to issues like repeated words or incomplete sentences because they don't explicitly model dependencies between output tokens. While iterative refinement methods exist to mitigate these problems, they can still be slow. This paper tackles the challenge of improving NAT's translation quality and reducing repetition without sacrificing its speed advantage.
Methods
The core innovation is the Local Autoregressive Translation (LAT) mechanism. Instead of predicting a single token at each decoding step, LAT predicts a short sequence of tokens (a "translation piece") autoregressively. This is done for each target position in parallel.
Key components of the LAT approach include:
- Local Translator: A lightweight, LSTM-based sequential decoder is used to generate these short "translation pieces" for each decoding position. The length of these pieces (denoted by K) is a hyperparameter, set to 3 in experiments to balance performance and speed.
- Efficient Merging Algorithm: Since LAT produces multiple overlapping "pieces," a novel algorithm is introduced to align and merge these pieces into a coherent final translation. This algorithm uses the Longest Common Subsequence (LCS) to find matching tokens between adjacent pieces and resolves conflicts by comparing confidence scores. It performs a local merge, considering only the last K tokens of the first piece and the first K tokens of the second piece to maintain efficiency and avoid global alignment issues.
- Integration with CMLM: The LAT mechanism is integrated into the Conditional Masked Language Model (CMLM) framework, which itself uses iterative decoding. This means the LAT-generated pieces are merged, and then the resulting sequence is refined over multiple iterations, with low-confidence tokens masked and re-predicted.
Datasets
The authors evaluated their method on five translation tasks:
- WMT'14 EN↔DE
- WMT'16 EN↔RO
- IWSLT'14 DE→EN
The paper does not introduce new benchmarks.
Key Insights and Ideas
- Local Dependencies are Key: The paper demonstrates that explicitly modeling local dependencies within short, autoregressively generated sequences is crucial for improving NAT quality.
- Merging is Efficient and Effective: The proposed merging algorithm is surprisingly effective at combining these local pieces, leveraging overlaps as alignment points and resolving conflicts intelligently. It's also computationally efficient, adding minimal overhead.
- Iterative Refinement Benefits from LAT: Integrating LAT into an iterative decoding framework like CMLM allows for faster convergence to good translation quality, requiring fewer iterations than standard CMLM.
- Reduced Repetition: The merging process naturally helps to mitigate the problem of repeated translations, a common issue in NAT.
Key Results
- Improved BLEU Scores: LAT consistently achieved comparable or better BLEU scores than CMLM at the same number of decoding iterations. For instance, LAT with 4 iterations performed similarly to CMLM with 10 iterations.
- Significant Speedup: The most striking result is the speedup. LAT achieved a nearly 2.5x speedup compared to CMLM with fewer decoding iterations. The average decoding latency was significantly reduced.
- Reduced N-gram Repeats: Table 2 shows that LAT significantly reduces n-gram repeat rates compared to CMLM, especially in early iterations.
- Better Performance on Longer Sentences: Figure 3 indicates that LAT performs better than CMLM on longer sentences, suggesting its ability to better capture longer-range dependencies through the iterative merging process.
Limitations and Future Work
- Hyperparameter Sensitivity (K): While the paper explores the effect of K (local translation steps), it notes that increasing K can improve BLEU but at the cost of inference time. The optimal K might vary across tasks.
- Length Adjustment: For intermediate iterations, a length adjustment procedure is used to match the predicted length. While effective, this adds complexity. The paper mentions an appendix for more details, suggesting this is a nuanced aspect.
- Future Work: The paper doesn't explicitly detail future work directions, but the success of LAT suggests potential for further exploration in other NAT architectures and for other sequence generation tasks beyond translation. The efficiency and quality improvements open doors for more practical NAT applications.
Summary updated