Abstract: Pedro Henrique Martins, João Alves, Tânia Vaz, Madalena Gonçalves, Beatriz Silva, Marianna Buchicchio, José G. C. de Souza, André F. T. Martins. Proceedings of the 24th Annual Conference of the European Association for Machine Translation.
2023.
Introduction
Modern Neural Machine Translation (NMT) systems typically operate as "fully parametric" models, where all acquired knowledge is stored within the model's weights. While effective, these systems often struggle with rare vocabulary, domain shifts, and "data drift" over time. To address these limitations, $k$-Nearest Neighbor Machine Translation ($k$NN-MT) was developed as a retrieval-augmented alternative. By accessing an external "datastore" of parallel sentences at inference time, $k$NN-MT allows models to supplement their internal logic with specific, retrieved examples.
This paper provides a critical evaluation of $k$NN-MT in production-like environments, moving beyond standard academic benchmarks to assess its performance in messy, real-world customer service scenarios.
Methodology
The $k$NN-MT approach augments a base NMT model (in this study, mBART50) with a token-level retrieval mechanism. During translation, the model generates a representation of the current context and queries a datastore for the $k$ most similar entries.
The final prediction is an interpolation between two probability distributions: the distribution from the base model's internal logic ($p_{\text{NMT}}$) and the distribution derived from the retrieved neighbors in the datastore ($p_{k\text{NN}}$). A hyperparameter, $\lambda$, controls the balance between these two sources, allowing the system to dynamically weight the retrieved memory against the model's learned parameters.
Experimental Setup
The authors evaluated four strategies: a base mBART50 model, $k$NN-MT, Full Fine-tuning (610M parameters), and Adapter-based Fine-tuning (~2% parameters). Testing was conducted across five proprietary datasets in four language pairs (En-Tr, En-Ko, En-De, En-Fr), covering press releases and customer service chat logs. These datasets ranged in size from 10,000 to over 1.3 million sentences, providing a diverse testbed for retrieval-augmented translation.
Results and Discussion
The Metric Gap and Human Preference
A significant finding of this study is the discrepancy between automatic metrics and human judgment. While BLEU and COMET scores often favored fine-tuned models, expert human evaluation (MQM) revealed that $k$NN-MT frequently outperformed them. In two out of three datasets, humans preferred $k$NN-MT, noting its superior ability to handle noisy source text—such as typos and grammatical errors common in customer support—by retrieving "cleaner" examples from the datastore.
Scaling and Efficiency
The research highlights that massive datastores are not a prerequisite for significant quality improvements. Even with relatively small datastores, $k$NN-MT provides a substantial performance boost over the base model.
Furthermore, the authors challenged the assumption that the entire datastore is needed to train the retrieval index (FAISS). They found that "bootstrapping" the index with as few as 10,000 entries was sufficient to reach peak retrieval performance, significantly simplifying the deployment pipeline.
Error Reduction
MQM analysis showed that $k$NN-MT is particularly effective at reducing "critical" and "major" translation errors. In the English-German customer service task, $k$NN-MT achieved an MQM score of 86.22, surpassing the fully fine-tuned model's 85.03.
Conclusion
While $k$NN-MT demonstrates remarkable robustness and high human-rated quality in real-world scenarios, it faces a significant challenge in inference latency. The requirement for a similarity search at every token generation step makes it considerably slower than traditional models. Future research should prioritize efficiency—through techniques like datastore pruning or phrase-level retrieval—to make this powerful retrieval-augmented approach viable for high-throughput production environments.
Summary updated