TL;DR
This paper introduces an enhanced Quality Estimation (QE) method for Machine Translation (MT) by incorporating "pseudo-references" – machine-translated versions of the source text – as an additional data source. This approach, building on previous kernel-based QE methods, demonstrates improved performance, particularly for Neural Machine Translation (NMT) systems, suggesting that even imperfect synthetic data can boost MT quality assessment.
What Problem Does This Paper Address?
Evaluating the quality of Machine Translation (MT) output without human references is a crucial task for efficient MT system development. Traditional methods often rely on comparing MT output to human-created "gold standard" translations, which are expensive and time-consuming to produce. Quality Estimation (QE) aims to predict translation quality automatically, often by learning from features extracted from the source text, the MT output, and sometimes back-translations. However, existing QE models can still struggle to accurately capture nuanced quality aspects, especially for newer MT paradigms like NMT. This paper addresses the challenge of further improving QE accuracy by exploring the utility of synthetically generated translations (pseudo-references) as an additional source of information for kernel-based QE models.
Methods
The core of this work is an extension of a previously developed Tree and Sequence Kernel Quality Estimation (TSKQE) method. The original TSKQE model used tree kernels (specifically, the Partial Tree Kernel) and sequence kernels (Subsequence Kernel) applied to the source segments, candidate translations, and back-translations of the MT output.
The novel contribution here is the incorporation of pseudo-references as an additional input to these kernel functions. Pseudo-references are machine translations of the source segments into the target language, generated using the same online MT system as the back-translations. By applying both sequence and tree kernels to this new data source, the authors investigate whether this artificially generated information can enhance the QE model's ability to predict translation quality. The paper uses a specific notation to denote which data sources (source, basic (source+candidate), back-translation, pseudo-reference) and which kernel types (sequence kernel 'sk', partial tree kernel 'ptk', or both) are used for each variant of TSKQE.
Datasets
The evaluation was performed on datasets provided by the WMT18 Quality Estimation sentence level task for the English-German language pair. These datasets included:
- SMT track: 26,273 instances for training and 1,000 for evaluation.
- NMT track: 13,442 instances for training and 1,000 for evaluation.
Each data tuple consisted of the source segment, the candidate translation, the reference translation, and the Human-mediated Translation Edit Rate (HTER) score, which served as the gold standard for quality.
Key Insights and Ideas
- Pseudo-references can boost QE performance, especially for NMT: The most significant finding is that incorporating pseudo-references led to noticeable improvements in QE accuracy for NMT systems, suggesting that even imperfect synthetic data can be valuable.
- Sequence kernels are robust: The paper highlights that sequence kernel-based variants performed well, even for language pairs that might lack robust syntactic analysis tools, indicating their general applicability.
- TSKQE is robust across MT paradigms: The method showed comparable results for both SMT and NMT systems, demonstrating its adaptability.
- Kernel-based QE outperforms feature-based baselines: The TSKQE models consistently outperformed the weak baseline (feature-based) system and, in most cases, the strong baseline (kernel-based without pseudo-references), reinforcing the effectiveness of kernel methods.
Key Results
The TSKQE variants generally outperformed the baseline systems in terms of Pearson correlation. Specifically:
- The best performing model across both SMT and NMT systems was TSKQE<sub>basic+back+pseudo</sub>, which utilized all data sources including pseudo-references.
- For the NMT system, the improvements gained by using pseudo-references were statistically significant for the
basic+back+pseudomodels over thebasic+backones at a 0.05 level. - In the SMT system, the impact of pseudo-references was less pronounced, with no statistically significant difference observed between
basic+back+pseudoandbasic+backmodels. - The paper ranked 4th and 6th out of 13 systems in the SMT track and 4th and 5th out of 11 systems in the NMT track of the WMT18 QE task.
Limitations and Future Work
- SMT vs. NMT performance difference: The authors note that the benefit of pseudo-references was more pronounced in NMT than in SMT. Further investigation into why this difference exists is warranted.
- Future work directions: The authors suggest extending the evaluation to more language pairs and domains. They also propose exploring the use of constituency trees in addition to the dependency trees already used to see if these additional syntactic structures can further improve QE performance.
Summary updated