Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison

Maxime Bouthors, Josep Maria Crego, François YvonNAACL 2024 · DOI

TL;DR

Retrieval Augmented Neural Machine Translation (RAMT) systems use past translations to improve current ones, but the retrieval part is often overlooked. This paper systematically compares different retrieval methods with various RAMT architectures, revealing that the retrieval strategy significantly impacts translation quality, especially for non-autoregressive models. The findings emphasize the importance of "in-domain" retrieval and suggest simplifying the pipeline by removing filtering steps can improve performance.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) models, while powerful, often act as black boxes. Retrieval Augmented NMT (RAMT) aims to improve both performance and interpretability by leveraging a "translation memory" (TM) – a collection of previously translated sentence pairs. These TMs provide relevant examples to guide the translation process. However, most RAMT research focuses on how to use these retrieved examples, with less attention paid to the crucial retrieval step itself. This makes it difficult to compare different RAMT approaches and understand the interplay between retrieval and generation. This paper addresses this gap by systematically comparing various retrieval strategies across different RAMT architectures to understand their impact and provide recommendations.

Methods

The paper breaks down the retrieval pipeline into three main steps:

  1. Domain Selection: Choosing which part of the TM to search (e.g., only "in-domain" data).
  2. Filtering: Narrowing down the initial set of retrieved examples based on superficial similarity. Methods include n-gram matching (NGM) and BM25.
  3. Ranking: Scoring the filtered examples to select the most relevant ones. This involves metrics like edit distance (LED) and a novel smoothed Longest Common Subsequence (δ-LCS) that balances coverage, relevance, and length. They also explore "contrastive ranking" to encourage diversity among retrieved examples.

They then integrate these retrieval methods with three NMT architectures:

  • Neural Fuzzy Augmentation (NFA): An autoregressive approach.
  • TM-LevT: An edit-based, non-autoregressive model.
  • Large Language Models (LLMs) with In-Context Learning (ICL): Using BLOOM.

Datasets

The experiments were conducted on two translation directions: English-to-French (en-fr) and German-to-English (de-en). For en-fr, they used a variety of domains including ECB, EMEA, Europarl, GNOME, JRC-Acquis, KDE4, PHP, Ubuntu, and OpenSubtitles. For de-en, they used domains like IT, Kor, Law, Med, and Sub. The en-fr test sets were specifically constructed to have at least one close example, allowing for analysis of retrieval quality's impact on translation scores.

Key Insights and Ideas

  • Retrieval Matters, Especially for Some Architectures: While NFA models are surprisingly robust to retrieval variations, TM-LevT and ICL architectures show significant gains when retrieval is optimized. This highlights that the choice of retrieval strategy is not one-size-fits-all.
  • "In-Domain" is Usually Better: Restricting retrieval to examples from the same domain generally leads to better translation scores, even for smaller domains. This suggests that contextually relevant examples are more valuable than a larger, but less relevant, pool.
  • Coverage is Key for Non-Autoregressive Models: The δ-LCS metric, which prioritizes covering more of the source sentence, shows particular benefits for TM-LevT, suggesting that non-autoregressive models can effectively leverage examples that provide broader source coverage.
  • Simplifying Inference Can Help: Removing the filtering step during inference (while keeping it for training) can improve scores for TM-LevT, albeit at the cost of increased latency. This points to a potential trade-off between speed and accuracy.
  • Diversity Can Be a Double-Edged Sword: Contrastive ranking, which promotes diversity, is beneficial when retrieval yields only medium-quality matches. However, it can be detrimental if high-quality matches are available, as it might discard them in favor of less relevant but diverse examples.

Key Results

  • Significant Gains for TM-LevT: By optimizing retrieval, TM-LevT achieved substantial improvements over baselines, particularly for the test-0.4 and de-en settings.
  • NFA Robustness: NFA models showed minimal performance variation across different retrieval settings, indicating their inherent resilience.
  • δ-LCS vs. LED: The δ-LCS ranking metric generally outperformed LED in terms of coverage and relevance, leading to better translation scores for TM-LevT and ICL.
  • Domain Selection Impact: "All-domain" or "out-of-domain" policies were detrimental for most architectures, with Ubuntu and News-Commentary being particularly affected.
  • Copy Rate Correlation: Higher copy rates (using parts of retrieved examples directly) correlated with higher BLEU scores for TM-LevT and ICL, but not for NFA.

Limitations and Future Work

The authors acknowledge several limitations:

  • Transductive Approach: All experiments used a frozen network, meaning no on-the-fly fine-tuning or adaptation was performed. Future work could explore these dynamic approaches.
  • Single LLM: Only one large LLM (BLOOM 176B) was used. Testing with newer, smaller, or different LLMs could provide a more comprehensive picture.
  • Focus on Translation Quality: The discussion primarily focused on translation metrics, with less emphasis on the computational costs and performance trade-offs of different retrieval/architecture combinations.

Future work could involve jointly optimizing retrieval and translation processes rather than treating them independently.

Summary updated

BibTeX
@inproceedings{bouthors2024retrieving,
  author = {Maxime Bouthors and Josep Maria Crego and François Yvon},
  title = {{Retrieving Examples from Memory for Retrieval Augmented Neural Machine Translation: A Systematic Comparison}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.190},
}
Figure