Introduction
In Machine Translation (MT), evaluating output quality typically relies on reference-based metrics like BLEU. However, real-world applications often lack these "gold standard" references. Quality Estimation (QE) addresses this by predicting translation quality using only the source text and the MT output. Despite its utility, QE development faces the "QE Paradox": training effective models requires large-scale, human-annotated post-edited datasets, which are prohibitively expensive and scarce for language pairs like Korean-English.
This study proposes three automated strategies to generate synthetic "pseudo-QE" datasets, bypassing the need for human labor and enabling the development of robust word-level QE models for Korean-English NMT.
Methodology
The researchers developed three distinct strategies to create synthetic training data from existing corpora and external translation engines:
- Monolingual-based (M-based): This strategy utilizes only target-language (English) text. It employs a "round-trip" translation process (English $\rightarrow$ Korean $\rightarrow$ English). The original English text serves as the reference, while the round-trip result acts as the flawed MT output.
- Parallel-based (P-based): This approach uses a standard parallel corpus (Korean-English). The Korean source is translated into English via an MT engine, and the result is compared against the existing human-translated English reference.
- Hybrid-based (H-based): A combination of the two, using the Korean source from a parallel corpus but comparing the MT output against a round-trip translation of the target side.
Labeling and Modeling
To generate word-level labels (OK vs. BAD), the Tercom tool was used to calculate the Levenshtein edit distance between the MT output and the reference. Words requiring deletion or substitution were tagged as BAD.
The models were built by fine-tuning various Multilingual Pretrained Language Models (mPLMs), including XLM, XLM-RoBERTa (XLM-R), and mBART, within the TransQuest framework. The input sequence followed the format: [CLS] Source [SEP] MT_Output_with_Gap_Tokens.
Experimental Setup
The models were trained and validated on 96K and 12K sentences, respectively, sourced from the AI HUB Korean-English parallel corpus. To evaluate the model's objectivity and generalization, it was tested against 12K sentences generated by four different commercial translation engines: Google, Amazon, Microsoft, and Systran.
Results and Discussion
The experiments yielded several key insights:
- Superiority of XLM-R: XLM-R-large consistently outperformed other mPLMs. This is attributed to its training on the massive CommonCrawl dataset, providing a more comprehensive representation of the Korean language compared to Wikipedia-based models.
- Effectiveness of Synthetic Data: The P-based strategy proved most effective, achieving a Matthew Correlation Coefficient (MCC) of 0.401 on the Google test set. However, the M-based strategy (MCC 0.364) remains a viable alternative for low-resource scenarios where parallel data is unavailable.
- Cross-Engine Objectivity: A significant finding was the model's stability across different translation engines. Despite being trained on data from a single engine, the performance variance across Google, Amazon, and Microsoft was minimal. This indicates that the model successfully learned general characteristics of translation errors rather than engine-specific biases.
Conclusion
This research demonstrates that high-quality word-level QE models for Korean-English NMT can be developed using synthetic data generation strategies. By leveraging mPLMs like XLM-R, these models provide an objective means of assessing translation quality without the need for expensive human post-editing. Future work will focus on scaling the pseudo-QE datasets and improving the prediction of "Gap" tags to better identify missing words in translations.
Summary updated