This paper explores how to make Non-Autoregressive Machine Translation (NAT) faster and more accurate by leveraging Translation Memories (TM)—databases of previously translated segments. The authors introduce TM-LevT, a modified Levenshtein Transformer that effectively "edits" retrieved translation matches to produce high-quality results at high speeds.
TL;DR
TM-LevT integrates Translation Memories into a non-autoregressive framework by adding a specialized deletion step and providing the model with "hints" on the source side. It achieves translation quality comparable to standard autoregressive models while being over 2x faster and—crucially—eliminating the need for the complex "knowledge distillation" step usually required for NAT.
What Problem Does This Paper Address?
Machine translation generally falls into two camps: Autoregressive (AR) models, which generate words one by one (high quality, slow), and Non-Autoregressive (NAT) models, which generate words in parallel (fast, but often lower quality).
NAT models struggle with the "multimodality problem": because they predict all words at once, they might mix and match parts of two different valid translations, resulting in a garbled mess. To fix this, researchers usually use Knowledge Distillation (KD), where a NAT model is trained on simplified outputs from an AR "teacher" model. However, this simplification can lose the nuance of real human language.
The authors noticed that the Levenshtein Transformer (LevT)—a NAT model that translates by iteratively inserting and deleting tokens—should be perfect for using Translation Memories. In professional translation, a TM provides a "fuzzy match" (a similar past translation) that a human then edits. However, vanilla LevT fails at this because it is trained to delete its own prediction errors, not the "unrelated" words found in a fuzzy TM match.
Methods
The authors propose TM-LevT, which introduces three key innovations:
- Initial Deletion (
init-del): As shown in Figure 1, the authors add a specific deletion step at the very beginning of the process. This step is trained specifically to identify and remove tokens in the TM match that are irrelevant to the current source sentence. - Source-Side Augmentation: The retrieved TM match is concatenated to the source sentence in the encoder. This ensures the model always has a "memory" of the original hint, preventing it from accidentally losing useful information during the iterative editing process.
- Multi-task Training: The model is trained to handle two scenarios simultaneously: translating from scratch (starting with an empty sequence) and translating from a TM match. This makes the model robust even when no good match is found in the database.
Datasets
The researchers evaluated their model on a large-scale multi-domain English-French corpus containing 11 different domains, ranging from technical manuals (GNOME, Ubuntu, PHP) and legal texts (JRC) to more general content like TED talks and Wikipedia. The final training set consisted of 4.4M parallel sentences, augmented with roughly 5M examples that included TM matches.
Key Insights and Ideas
- The "Fluent but Inadequate" Trap: The authors discovered that vanilla LevT is easily "fooled" by TM matches. Because a TM match is usually a fluent sentence, the original LevT deletion module (trained only on its own messy intermediate predictions) doesn't realize it needs to change anything, even if the words are wrong for the current context.
- TMs Solve Multimodality: One of the most significant findings is that using a TM match provides enough explicit context to resolve the multimodality problem. This means TM-LevT can be trained on real human translations rather than distilled data, leading to better performance. In fact, the authors found that applying Knowledge Distillation actually hurt performance when using TMs.
- Graceful Degradation: As shown in Figure 3, the model is highly adaptive. When the TM match is excellent (high similarity), it relies heavily on it. When the match is poor, it ignores the match and performs as well as a standard MT system.
Key Results
- Quality: TM-LevT achieves BLEU scores within 1.2 to 2.4 points of strong AR models. On specific domains like Wikipedia, it even outperformed the AR baseline.
- Speed: TM-LevT is significantly faster, providing a 2.28x to 2.34x speedup in decoding time compared to AR models.
- Efficiency: Starting from a TM match reduces the number of iterations the model needs to converge by about 20% compared to starting from scratch.
Limitations and Future Work
- Computational Overhead: The extra deletion step and dual-task training increase training time by about 10-20% compared to the basic LevT.
- Language Diversity: The study was limited to the English-French language pair. The authors acknowledge that testing on more "distant" language pairs (like English-Chinese) is necessary to see if these benefits hold up when sentence structures differ more radically.
- Retrieval Dependency: The system's peak performance depends on having a high-quality TM. While it handles poor matches well, its primary advantage is realized in domains with high repetitive content.
Reference: Xu, J., Crego, J., & Yvon, F. (2023). Integrating Translation Memories into Non-Autoregressive Machine Translation. arXiv:2210.06020v2 [cs.CL].
Summary updated