Multilingual \textit{k}-Nearest-Neighbor Machine Translation

David Stap, Christof MonzEMNLP 2023 · OpenReview

TL;DR

This paper introduces multilingual k-Nearest-Neighbor Machine Translation (kNN-MT) to improve translation quality for low-resource languages. By creating unified datastores from multiple languages and leveraging linguistic similarities, they achieve significant BLEU score improvements and drastically reduce inference time compared to monolithic datastores.

What Problem Does This Paper Address?

Traditional kNN-MT methods rely on large, single-language datastores to enhance translation quality by retrieving similar translation examples. However, this approach has been largely ineffective for low-resource languages due to data scarcity and the need for massive datastores. The authors aim to bridge this gap by exploring how multilingual models (mNMT) can be used to build more effective and efficient kNN-MT systems, particularly for languages with limited parallel data. They also address the computational cost associated with large datastores, which hinders practical deployment.

Methods

The core innovation is the creation of multilingual datastores. Instead of separate datastores for each language pair, the authors merge bilingual datastores from related languages into a single, unified datastore. They also explore using representations from multilingual neural machine translation (mNMT) models to generate these datastores.

A key aspect of their approach is leveraging linguistic similarities to construct smaller, yet effective, multilingual datastores. This involves grouping languages by family (e.g., Slavic, Germanic) and creating datastores based on these groupings. They also investigate a cross-lingual mapping technique to align representations from different languages, aiming to improve retrieval accuracy.

Datasets

The experiments are conducted on the TED Talks corpus, a widely used dataset for machine translation research. The authors utilize 51 languages from the M2M100 multilingual translation model, focusing on translations into English. They create datastores from the training set, tune hyperparameters on the development set, and report results on the test set.

Key Insights and Ideas

  • Multilingual datastores are a game-changer for low-resource languages: The paper demonstrates that unifying datastores across related languages significantly boosts translation quality for low-resource pairs, a feat difficult to achieve with traditional bilingual datastores.
  • Linguistic similarity is key to efficiency: Grouping languages by family and constructing datastores based on these groups allows for substantial reductions in datastore size without sacrificing translation quality, leading to significant speedups.
  • Cross-lingual retrieval is more nuanced than just size: While larger cross-lingual datastores generally help, the effectiveness also depends on the linguistic relatedness between the source language of the datastore and the target language of the translation task.
  • mNMT representations generalize well: The use of mNMT models proves effective in generating representations that allow for meaningful cross-lingual retrieval, even for languages not directly present in the original bilingual datastores.

Key Results

  • Substantial improvements for low-resource languages: The proposed multilingual kNN-MT approach achieves up to +3.6 BLEU improvement for low-resource translation quality.
  • Improvements for high-resource languages too: Even high-resource languages see gains, with up to +0.5 BLEU improvement.
  • Significant speedups: By creating multilingual datastores that are a quarter of the size of monolithic ones, the authors achieve a 5.3x speed improvement in decoding time. For example, using a Slavic language grouping datastore for Belarusian-English translation resulted in a 5.3x speedup.
  • Effective datastore size reduction: Datastores built on linguistic similarity (e.g., Slavic grouping) were significantly smaller (e.g., 30.6M entries) than the comprehensive multilingual datastore (125M entries) while maintaining comparable or even better performance for some language pairs.

Limitations and Future Work

  • English-centric bias: The current mNMT model used is not strictly English-centric, but the authors acknowledge that results might differ in heavily English-centric multilingual settings.
  • Scalability of very large datastores: While they demonstrate efficiency gains with smaller, linguistically grouped datastores, the paper does not explore the potential benefits of even larger multilingual datastores beyond 51 languages.
  • Predicting optimal cross-lingual datastores: The authors note that it is difficult to predict which specific cross-lingual datastore will yield the best results for a given language pair, suggesting further research into this predictability.
  • Future work could involve exploring:
    • The impact of even larger multilingual datastores.
  • More sophisticated methods for aligning multilingual representations.
  • Investigating the performance of this approach on non-English centric translation directions.

Summary updated

Abstract

\textit{k}-nearest-neighbor machine translation has demonstrated remarkable improvements in machine translation quality by creating a datastore of cached examples. However, these improvements have been limited to high-resource language pairs, with large datastores, and remain a challenge for low-resource languages. In this paper, we address this issue by combining representations from multiple languages into a single datastore. Our results consistently demonstrate substantial improvements not only in low-resource translation quality (up to $+3.6$ BLEU), but also for high-resource translation quality (up to $+0.5$ BLEU). Our experiments show that it is possible to create multilingual datastores that are a quarter of the size, achieving a 5.3x speed improvement, by using linguistic similarities for datastore creation.\footnote{We will release our code upon acceptance.}
BibTeX
@inproceedings{stap2023multilingual,
  author = {David Stap and Christof Monz},
  title = {{Multilingual \textbackslash\{\}textit\{k\}-Nearest-Neighbor Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=RKqtOoMC1M},
}
Figure