Fixing Rogue Memorization in Many-to-One Multilingual Translators of Extremely-Low-Resource Languages by Rephrasing Training Samples

Paulo R. Cavalin, Pedro Henrique Domingues, Claudio S. Pinhanez, Julio NogimaNAACL 2024 · DOI

TL;DR

This paper investigates a critical issue in fine-tuning large language models (LLMs) for extremely low-resource languages: "rogue memorization," where models regurgitate training data instead of translating. The authors demonstrate that rephrasing training samples effectively mitigates this problem, leading to improved translation quality and reduced memorization without sacrificing performance.

What Problem Does This Paper Address?

The standard approach to building machine translators for low-resource languages involves fine-tuning pre-trained, high-resource LLMs on limited parallel data. While this generally improves accuracy over training from scratch, this paper highlights a significant pitfall: "rogue memorization." This occurs when many-to-one multilingual models, especially those trained on diverse languages, learn to simply store and retrieve output strings from the training data rather than performing actual translations. This leads to inflated performance metrics (like BLEU scores) that mask a fundamental failure of the model to generalize, making it unsuitable for real-world translation tasks. The paper aims to identify, quantify, and solve this memorization issue in the context of translating endangered Indigenous languages.

Methods

The researchers employed two main LLMs: mBART50 and WMT19. They experimented with three fine-tuning strategies:

  • Bilingual (BL): Fine-tuning on data from a single language pair.
  • All Languages (AL): Fine-tuning on data from all 39 languages simultaneously.
  • Tupi-Family (TF): Fine-tuning on a subset of 10 related languages.

To quantify memorization, they developed a novel metric based on the minimum Euclidean distance between a generated output and any sample in the training set, using sentence embeddings. This approach captures "soft" memorization, going beyond exact matches. They also analyzed the distribution of BLEU scores and the number of distinct outputs to identify memorization patterns. The core proposed solution involves rephrasing the target output sentences in the training data. Instead of using identical verses from different Bible translations, they used human-generated rephrased versions to provide semantically similar but not identical training examples.

Datasets

The primary dataset used is "The Bibles" dataset, a collection of parallel verses from the New Testament in English and 39 Brazilian Indigenous Languages (BILs). This dataset was compiled from online Bible resources. For evaluation, the New Testament's Matthew chapter was held out as a test set, with the remainder used for training. The paper also mentions using multiple English Bible versions for rephrasing.

Key Insights and Ideas

  • Metric Misleadingness: Standard translation metrics like BLEU can be deceptive in low-resource scenarios, masking "rogue memorization" by rewarding perfectly recalled training samples.
  • Many-to-One Amplifies Memorization: The "many-to-one" nature of multilingual translation, especially with more languages, significantly increases the tendency for models to memorize training data.
  • Standard Deviations as Indicators: High standard deviations in translation metrics are a strong symptom of memorization issues.
  • Rephrasing as a Solution: Replacing identical training outputs with semantically similar but distinct rephrased versions effectively combats memorization without harming translation quality.
  • Memorization in Decoders: The study suggests that memorization might occur primarily on the decoder side of Transformer architectures, leading to a disconnection from the encoder's input.

Key Results

  • Quantified Memorization: The AL39 models (mBART50-AL39 and WMT19-AL39) showed significantly lower memorization metric scores (mean distances of 0.14-0.17) compared to bilingual models (0.62-0.66), indicating strong memorization.
  • Distinct Outputs: The mBART50-AL39 model produced only 15% of the expected distinct outputs, mapping many inputs to the same memorized output (a 7:1 ratio).
  • Rephrasing Impact: After rephrasing, the mBART50-TF10 model showed a dramatic reduction in memorization metric scores (from 0.47 to 0.90 for training samples, and 0.61 for test samples).
  • Improved Quality (with caveats): Rephrasing led to gains of 6% in BLEURT and 1% in BERTScore, but a 15% loss in BLEU. Crucially, standard deviations for all metrics decreased significantly, suggesting more stable and reliable translations.
  • Distinct Outputs with Rephrasing: The rephrased model generated 99.5% of the maximum possible distinct outputs, far exceeding other models tested.

Limitations and Future Work

  • Dataset Domain: The use of "The Bibles" dataset is a limitation due to its specific domain and potential overlap with pre-training data. The authors acknowledge ethical considerations and the "toxic" nature of such data, emphasizing its use for controlled laboratory experiments.
  • LLM-Generated Rephrasing: While human rephrased versions were used, the paper suggests that LLM-generated rephrasings could be a viable alternative, and evaluating their success in mitigating memorization is a future research direction.
  • Broader Applicability: The authors hypothesize that similar memorization issues might occur in other tasks, such as fine-tuning conversational LLMs for question answering, and suggest this as a future research avenue.
  • Metric-Based Model Selection: The proposed memorization metric could be used for model selection or early stopping, but its impact on translation accuracy needs further investigation.

Summary updated

BibTeX
@inproceedings{cavalin2024fixing,
  author = {Paulo R. Cavalin and Pedro Henrique Domingues and Claudio S. Pinhanez and Julio Nogima},
  title = {{Fixing Rogue Memorization in Many-to-One Multilingual Translators of Extremely-Low-Resource Languages by Rephrasing Training Samples}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.253},
}
Figure