Abstract: Quality Estimation (QE) models have the potential to change how we evaluate and maybe even train machine translation models. However, these models still lack the robustness to achieve general adoption. We show that State-of-the-art QE models, when tested in a Parallel Corpus Mining (PCM) setting, perform unexpectedly bad due to a lack of robustness to out-of-domain examples. We propose a combination of multitask training, data augmentation and contrastive learning to achieve better and more robust QE performance. We show that our method improves QE performance significantly in the MLQE challenge and the robustness of QE models when tested in the Parallel Corpus Mining setup. We increase the accuracy in PCM by more than 0.80, making it on par with state-of-the-art PCM methods that use millions of sentence pairs to train their models. In comparison, we use a thousand times less data, 7K parallel sentences in total, and propose a novel low resource PCM method.
Introduction
In Machine Translation (MT), Quality Estimation (QE) is the task of predicting the quality of a translation without a reference answer. This capability is critical for Parallel Corpus Mining (PCM), where researchers scour the web to find matching sentence pairs to train new MT models. However, current QE models suffer from a significant robustness gap. They are often "fooled" by fluency, assigning high scores to grammatically correct sentence pairs that have entirely different meanings. This lack of semantic grounding causes them to fail in real-world corpus mining scenarios.
Furthermore, high-performing PCM tools typically require millions of parallel sentences for training—a luxury unavailable for low-resource languages. This research addresses this "paradoxical loop" by introducing MultiQE, a framework that achieves state-of-the-art performance using 1,000x less data than standard methods.
Methodology
The MultiQE framework leverages a single backbone model (XLM-R) and improves its semantic understanding through three primary innovations:
- Multitask Learning: The model is trained simultaneously on QE, Semantic Textual Similarity (STS), and Natural Language Inference (NLI). This forces the model to learn universal representations of meaning that transfer across languages.
- Negative Data Augmentation (DA): To combat the "fluency trap," the model is exposed to "bad" pairs—sentences that are fluent but semantically unrelated. This teaches the model that high quality requires both grammatical correctness and semantic alignment.
- Contrastive Learning: A contrastive loss function is used to train a filtration model, allowing for the rapid narrowing of millions of potential sentence pairs into a manageable candidate pool before expensive QE scoring occurs.
Experimental Setup
The study utilizes several key benchmarks to evaluate performance and robustness:
- MLQE (Multilingual Quality Estimation): The primary training and testing dataset, including low-resource pairs like Nepali-English (ne-en) and Sinhala-English (si-en).
- STS-B & MNLI: Monolingual English datasets used for the multitask training phase.
- TATOEBA & BUCC: Standard benchmarks for Parallel Corpus Mining, used to test the models' ability to find matching translations in large-scale datasets.
Results and Discussion
MultiQE demonstrated significant improvements over established baselines. In the TATOEBA corpus mining challenge, the framework increased accuracy by more than 0.80 compared to standard QE models. Notably, on the Sinhala-English task, MultiQE outperformed LASER, a high-resource model trained on orders of magnitude more data.
As shown in the results, MultiQE Multitask achieved the highest Pearson correlation with human judgment on 4 out of 6 language pairs in the MLQE dataset. A key insight from this work is that training on English-only semantic tasks (STS and MNLI) significantly improves the model's ability to estimate quality in other languages, proving that the underlying logic of "meaning" transfers effectively across multilingual models.
Conclusion
MultiQE challenges the "big data" status quo in machine translation by proving that smarter training signals—specifically multitask learning and negative sampling—can compensate for a lack of massive parallel datasets. While the model performed exceptionally well on TATOEBA, its performance on the BUCC dataset suggests room for improvement, though manual inspection revealed that many "errors" were actually valid translations missing from the gold standard.
Future work will explore self-supervised contrastive learning to further reduce reliance on human-labeled scores and investigate the integration of QE directly into MT training loops as a real-time data curator.
Summary updated