Implementing and Evaluating Multi-source Retrieval-Augmented Translation

TL;DR

This paper introduces Contrastive Ensembling, a novel method for Retrieval-Augmented Translation (RAT) that allows NMT systems to simultaneously utilize multiple external knowledge sources, such as terminology databases and translation memories. By evaluating against a rigorous new English-to-Finnish test suite, the authors demonstrate that this approach effectively incorporates external constraints while maintaining high translation quality, even outperforming large language models (LLMs) in specific retrieval-utilization tasks.

What Problem Does This Paper Address?

Standard Neural Machine Translation (NMT) models typically translate sentences in isolation, often missing critical domain-specific context or failing to adhere to established terminology. Retrieval-Augmented Translation (RAT) attempts to solve this by querying external databases for relevant "fuzzy matches" (similar previously translated sentences) or specific "term matches."

However, most existing RAT research focuses on integrating only one type of information at a time. In professional translation workflows, human translators use both simultaneously. Furthermore, standard evaluation metrics like BLEU or COMET measure general quality but fail to capture whether a model actually used the retrieved information. This paper tackles the challenge of multi-source RAT and the lack of granular evaluation benchmarks.

Methods

The authors experiment with two main architectures:

1. Unified Models: A single NMT model trained on data augmented with both terms and fuzzy matches.
2. Multisource Ensembling: Combining specialized models (e.g., one trained for terms, one for fuzzy matches).

The core innovation is Contrastive Ensembling. In a "naive" ensemble, the probabilities from different models are simply averaged, which often dilutes the influence of the specialized models. Contrastive ensembling (illustrated in Figure 2) addresses this by comparing the next-token probability distribution of a specialized RAT model against a "base" model that received no external info. If a token's probability is significantly higher in the RAT model, its weight is boosted in the final output.

Expand

The inference pipeline for this ensembling approach is shown in Figure 1, where shared output tokens are fed back into multiple models with different inputs.

Expand

Datasets

• Training: The models were fine-tuned on a 10-million-sentence subset of the Tatoeba-Challenge dataset.
• Evaluation: The authors created a new, purpose-built English-to-Finnish test suite. Unlike standard benchmarks, this suite includes specific "Include" and "Do not include" sets for fuzzy matches and regex-based tests for terminology to verify correct inflection in the target language (Finnish).

Expand

Key Insights and Ideas

• The Dilution Problem: Naive ensembling often results in the model ignoring external constraints because the "base" translation signal (which doesn't know about the terms) washes out the specialized signal.
• LLMs vs. NMT-RAT: While instruction-tuned LLMs like Gemma 3 are excellent at following terminology constraints, they are surprisingly less effective at utilizing fuzzy matches compared to specialized NMT-RAT models.
• Efficiency Matters: Specialized NMT-RAT models are significantly smaller and faster than the 9B-12B parameter LLMs tested, making them more suitable for local, low-latency deployment in professional translation tools (CAT tools).

Key Results

• Gemma 3 12B-IT achieved the highest overall "Composite Score" (0.671), largely due to its superior terminology accuracy (0.811).
• Contrastive Ensembling (ContrastEnsembleTS) significantly outperformed naive ensembling, raising the Term Score from 0.391 to 0.558 and the Fuzzy Score from 0.455 to 0.500.
• General Quality: Reference-free COMET scores remained stable across RAT methods, indicating that forcing the model to use external info does not degrade the overall naturalness of the translation.

Limitations and Future Work

The primary limitation is that the study and the new test suite are currently limited to a single language pair: English to Finnish. Finnish is morphologically complex, making it a challenging and useful test case, but results may vary for other language families.

Future work could involve expanding the test suite to more languages and exploring how to better prompt LLMs to utilize the structural information provided by fuzzy matches, where they currently lag behind specialized NMT architectures.

@inproceedings{nieminen2026implementing,
  author = {Tommi Nieminen and Jörg Tiedemann and Sami Virpioja},
  title = {{Implementing and Evaluating Multi-source Retrieval-Augmented Translation}},
  booktitle = {Proceedings of the Tenth Conference on Machine Translation},
  year = {2026},
  url = {https://openreview.net/forum?id=CCtd7qoPrh},
}