TL;DR
This paper introduces a novel approach to multilingual word-level Quality Estimation (QE) using cross-lingual transformers. They demonstrate that a single multilingual QE model can perform comparably to language-specific models, even in zero-shot and few-shot scenarios, making QE more accessible for low-resource languages.
What Problem Does This Paper Address?
Quality Estimation (QE) aims to assess the quality of machine translations without a reference. While sentence-level QE is common, word-level QE, which pinpoints specific errors at the word or gap level, offers more granular insights crucial for post-editors. However, existing word-level QE models are typically language-specific, requiring extensive labeled data and maintenance for each language pair. This is a significant bottleneck, especially for low-resource languages where data is scarce. The paper addresses this by exploring multilingual QE models that can generalize across languages, reducing the need for language-specific training data.
Methods
The core of the proposed method is an architecture built upon the XLM-R transformer model. XLM-R is pre-trained on a massive multilingual dataset covering 104 languages. The authors adapt this by adding a special <GAP> token to the target sentence to represent gaps between words. The source and target sentences, along with the <GAP> tokens, are then fed into XLM-R. A simple linear layer on top of the transformer's embeddings is used to predict the quality of each word and gap (labeled as "Good" or "Bad"). This architecture is then trained and evaluated in various multilingual settings, including supervised, zero-shot, and few-shot learning.
Datasets
The paper utilizes several language pairs for which word-level QE annotations were available. These include:
- English-Chinese (En-Zh)
- English-Czech (En-Cs)
- English-German (En-De)
- English-Russian (En-Ru)
- English-Latvian (En-Lv)
- German-English (De-En)
The translations for these pairs were produced using both neural and statistical machine translation systems, and the data comes from various domains. Specific details about the datasets, including the MT system and competition source, are provided in Table 1.
Key Insights and Ideas
- Multilingual Models Compete with Bilingual Models: A significant finding is that a single multilingual QE model can achieve performance comparable to, and sometimes even better than, models trained specifically for a single language pair. This challenges the necessity of language-specific models.
- Zero-Shot QE is Feasible: The paper demonstrates that multilingual QE models can be effectively used for zero-shot quality estimation on unseen language pairs. While performance might slightly decrease compared to bilingual models, it's often still competitive and significantly better than traditional baselines.
- Language Direction Matters for Zero-Shot: A notable limitation of zero-shot QE is its sensitivity to language direction. Models trained on source-to-target pairs struggle when tested on target-to-source pairs, a phenomenon observed in previous sentence-level QE research.
- Few-Shot Learning Benefits from Multilingual Pre-training: For scenarios with limited training data (few-shot learning), multilingual models significantly outperform models trained from scratch. This suggests that pre-training on a diverse set of languages provides valuable linguistic knowledge that accelerates learning for new tasks.
Key Results
The paper reports that their multilingual QE models perform "on par with the current language-specific models." Specifically, in zero-shot QE, the performance decrease compared to bilingual models was described as "negligible" for most pairs. For few-shot QE, the multilingual models consistently outperformed bilingual models trained from scratch, especially when training data was scarce. The authors also note that these findings hold true for target word F1-scores, target gap F1-scores, and source word F1-scores.
Limitations and Future Work
The authors acknowledge that zero-shot QE performance degrades when the language direction is reversed (e.g., training on De-En and testing on En-De). This suggests that while multilingual models generalize well, they are not entirely immune to language directionality. Future work could explore methods to mitigate this directionality issue. Additionally, the paper focuses on the F1-score for target words, but mentions that results for target gaps and source words are consistent, implying further analysis could be beneficial. The authors also suggest that their findings are particularly useful for low-resource languages where data scarcity is a major hurdle.
Summary updated