Integrating Translation Memories into Non-Autoregressive Machine Translation

Jitao Xu, Josep Crego, François Yvon

This paper explores how to make Non-Autoregressive Machine Translation (NAT) faster and more accurate by leveraging Translation Memories (TM)—databases of previously translated segments. The authors introduce TM-LevT, a modified Levenshtein Transformer that effectively "edits" retrieved translation matches to produce high-quality results at high speeds.

TL;DR

TM-LevT integrates Translation Memories into a non-autoregressive framework by adding a specialized deletion step and providing the model with "hints" on the source side. It achieves translation quality comparable to standard autoregressive models while being over 2x faster and—crucially—eliminating the need for the complex "knowledge distillation" step usually required for NAT.

What Problem Does This Paper Address?

Machine translation generally falls into two camps: Autoregressive (AR) models, which generate words one by one (high quality, slow), and Non-Autoregressive (NAT) models, which generate words in parallel (fast, but often lower quality).

NAT models struggle with the "multimodality problem": because they predict all words at once, they might mix and match parts of two different valid translations, resulting in a garbled mess. To fix this, researchers usually use Knowledge Distillation (KD), where a NAT model is trained on simplified outputs from an AR "teacher" model. However, this simplification can lose the nuance of real human language.

The authors noticed that the Levenshtein Transformer (LevT)—a NAT model that translates by iteratively inserting and deleting tokens—should be perfect for using Translation Memories. In professional translation, a TM provides a "fuzzy match" (a similar past translation) that a human then edits. However, vanilla LevT fails at this because it is trained to delete its own prediction errors, not the "unrelated" words found in a fuzzy TM match.

Methods

The authors propose TM-LevT, which introduces three key innovations:

  1. Initial Deletion (init-del): As shown in Figure 1, the authors add a specific deletion step at the very beginning of the process. This step is trained specifically to identify and remove tokens in the TM match that are irrelevant to the current source sentence.
  2. Source-Side Augmentation: The retrieved TM match is concatenated to the source sentence in the encoder. This ensures the model always has a "memory" of the original hint, preventing it from accidentally losing useful information during the iterative editing process.
  3. Multi-task Training: The model is trained to handle two scenarios simultaneously: translating from scratch (starting with an empty sequence) and translating from a TM match. This makes the model robust even when no good match is found in the database.
Figure 1: A complete training step for TM-LevT. Compared to the original LevT which starts training from e′′, TM-LevT adds the init-del step to delete unrelated tokens from a TM match. Figure better viewed in color.

Datasets

The researchers evaluated their model on a large-scale multi-domain English-French corpus containing 11 different domains, ranging from technical manuals (GNOME, Ubuntu, PHP) and legal texts (JRC) to more general content like TED talks and Wikipedia. The final training set consisted of 4.4M parallel sentences, augmented with roughly 5M examples that included TM matches.

Key Insights and Ideas

  • The "Fluent but Inadequate" Trap: The authors discovered that vanilla LevT is easily "fooled" by TM matches. Because a TM match is usually a fluent sentence, the original LevT deletion module (trained only on its own messy intermediate predictions) doesn't realize it needs to change anything, even if the words are wrong for the current context.
  • TMs Solve Multimodality: One of the most significant findings is that using a TM match provides enough explicit context to resolve the multimodality problem. This means TM-LevT can be trained on real human translations rather than distilled data, leading to better performance. In fact, the authors found that applying Knowledge Distillation actually hurt performance when using TMs.
  • Graceful Degradation: As shown in Figure 3, the model is highly adaptive. When the TM match is excellent (high similarity), it relies heavily on it. When the match is poor, it ignores the match and performs as well as a standard MT system.
Figure 3: BLEU scores when translating with TM matches of various similarity ranges. The Copy strat- egy uses the retrieved example as the final translation.

Key Results

  • Quality: TM-LevT achieves BLEU scores within 1.2 to 2.4 points of strong AR models. On specific domains like Wikipedia, it even outperformed the AR baseline.
  • Speed: TM-LevT is significantly faster, providing a 2.28x to 2.34x speedup in decoding time compared to AR models.
  • Efficiency: Starting from a TM match reduces the number of iterations the model needs to converge by about 20% compared to starting from scratch.

Limitations and Future Work

  • Computational Overhead: The extra deletion step and dual-task training increase training time by about 10-20% compared to the basic LevT.
  • Language Diversity: The study was limited to the English-French language pair. The authors acknowledge that testing on more "distant" language pairs (like English-Chinese) is necessary to see if these benefits hold up when sentence structures differ more radically.
  • Retrieval Dependency: The system's peak performance depends on having a high-quality TM. While it handles poor matches well, its primary advantage is realized in domains with high repetitive content.

Reference: Xu, J., Crego, J., & Yvon, F. (2023). Integrating Translation Memories into Non-Autoregressive Machine Translation. arXiv:2210.06020v2 [cs.CL].

Summary updated

Abstract

Non-autoregressive machine translation (NAT) has recently made great progress. However, most works to date have focused on standard translation tasks, even though some edit-based NAT models, such as the Levenshtein Transformer (LevT), seem well suited to translate with a Translation Memory (TM). This is the scenario considered here. We first analyze the vanilla LevT model and explain why it does not do well in this setting. We then propose a new variant, TM-LevT, and show how to effectively train this model. By modifying the data presentation and introducing an extra deletion operation, we obtain performance that are on par with an autoregressive approach, while reducing the decoding load. We also show that incorporating TMs during training dispenses to use knowledge distillation, a well-known trick used to mitigate the multimodality issue.
BibTeX
@inproceedings{xu2023integrating,
  author = {Jitao Xu and Josep Crego and François Yvon},
  title = {{Integrating Translation Memories into Non-Autoregressive Machine Translation}},
  booktitle = {EACL},
  year = {2023},
}
Figure