Abstract
Neural fuzzy repair (NFR) is a simple implementation of retrieval-augmented translation (RAT), based on data augmentation. In NFR, a translation database is searched for translation examples where the source sentence is similar to the sentence being translated, and the target side of the example is concatenated with the source sentences. We experiment with introducing retrieval that is based on target similarity to NFR during training. The results of our experiments confirm that including target similarity matches during training supplements source similarity matches and leads to better translations at translation time.
Introduction
In the field of Machine Translation (MT), Retrieval-Augmented Translation (RAT) provides models with a reference set of previous translations to guide current work. Neural Fuzzy Repair (NFR) is a streamlined implementation of this concept: when translating a sentence, the system retrieves a similar "source" sentence from a database and provides its corresponding "target" translation as a hint.
Traditionally, NFR retrieves these hints based on source-side similarity. However, similar source sentences do not always result in translations that share the specific sub-segments needed for the current task, creating a noisy training signal. This research explores whether providing the model with "ideal" examples during training—retrieved based on target-side similarity—can teach the model to become a more effective selective copier during inference.
Methodology
The authors experimented with several data augmentation strategies for an English-to-Finnish Transformer-based model. The core approach involves concatenating translation hints to the source sentence, separated by special tokens.
Key strategies included:
- Target-Similarity Retrieval (Trg-Sim): During training, examples are retrieved based on the similarity of the target side to the desired output, providing a clear signal of which segments should be copied.
- Hybrid Training (Combo & Mix-Sim): These methods combine source-similarity and target-similarity matches to prevent the model from becoming "lazy," forcing it to learn how to distinguish between relevant and irrelevant hints.
- Similarity Class Annotations: Matches are tagged with numerical suffixes (e.g., 5 to 9) to indicate the degree of similarity, helping the model decide how much to trust a specific hint.
Experimental Setup
The study utilized the Tatoeba-Challenge dataset for English-to-Finnish translation. A significant contribution of this work is the creation of Domeval and Domeval-NC, two new test sets designed specifically for evaluating retrieval-augmented systems. Unlike standard benchmarks, these sets ensure a high density of "fuzzy matches," allowing for a statistically significant measurement of the model's repair behavior.
Results and Discussion
The experimental results demonstrate that NFR models significantly outperform standard Transformer baselines. On the Domeval dataset, the baseline model achieved a BLEU score of 31.14, while the best-performing augmented model, 2X-Combo, reached 41.57 BLEU—a massive 10-point improvement.
Key insights from the evaluation include:
- Target Similarity as a "Teacher": Training with target-side matches acts as a beneficial form of "cheating" that teaches the model Sub-sentential Selection—the ability to identify which specific words within a hint to use.
- Inference Robustness: Models trained with target-side similarity performed better at test time (using only source-side retrieval) than models trained exclusively on source-side matches.
- Ensembling: The highest overall quality was achieved by ensembling source-similarity and target-similarity models, reaching a peak of 41.69 BLEU.
- Input Length Penalty: While additional hints generally help, performance eventually plateaus or degrades as the input length increases, suggesting a limit to the amount of context a model can effectively process via concatenation.
Conclusion
Incorporating target-side similarity during training significantly enhances the performance of Neural Fuzzy Repair systems. By providing a cleaner signal for sub-sentential copying, the model learns to better utilize retrieved translation hints. Future work will focus on scaling these methods to larger datasets and exploring non-concatenative architectures to handle multiple translation examples without the associated input length penalties.
Summary updated