Word-Level Quality Estimation for Korean-English Neural Machine Translation

Sugyeong Eo, Chanjun Park, Hyeonseok Moon, Jaehyung Seo, Heuiseok LimIEEE Access 2022 · DOI · OpenReview

Introduction

In Machine Translation (MT), evaluating output quality typically relies on reference-based metrics like BLEU. However, real-world applications often lack these "gold standard" references. Quality Estimation (QE) addresses this by predicting translation quality using only the source text and the MT output. Despite its utility, QE development faces the "QE Paradox": training effective models requires large-scale, human-annotated post-edited datasets, which are prohibitively expensive and scarce for language pairs like Korean-English.

This study proposes three automated strategies to generate synthetic "pseudo-QE" datasets, bypassing the need for human labor and enabling the development of robust word-level QE models for Korean-English NMT.

Methodology

The researchers developed three distinct strategies to create synthetic training data from existing corpora and external translation engines:

  1. Monolingual-based (M-based): This strategy utilizes only target-language (English) text. It employs a "round-trip" translation process (English $\rightarrow$ Korean $\rightarrow$ English). The original English text serves as the reference, while the round-trip result acts as the flawed MT output.
  2. Parallel-based (P-based): This approach uses a standard parallel corpus (Korean-English). The Korean source is translated into English via an MT engine, and the result is compared against the existing human-translated English reference.
  3. Hybrid-based (H-based): A combination of the two, using the Korean source from a parallel corpus but comparing the MT output against a round-trip translation of the target side.

Labeling and Modeling

To generate word-level labels (OK vs. BAD), the Tercom tool was used to calculate the Levenshtein edit distance between the MT output and the reference. Words requiring deletion or substitution were tagged as BAD.

The models were built by fine-tuning various Multilingual Pretrained Language Models (mPLMs), including XLM, XLM-RoBERTa (XLM-R), and mBART, within the TransQuest framework. The input sequence followed the format: [CLS] Source [SEP] MT_Output_with_Gap_Tokens.

Experimental Setup

The models were trained and validated on 96K and 12K sentences, respectively, sourced from the AI HUB Korean-English parallel corpus. To evaluate the model's objectivity and generalization, it was tested against 12K sentences generated by four different commercial translation engines: Google, Amazon, Microsoft, and Systran.

Results and Discussion

The experiments yielded several key insights:

  • Superiority of XLM-R: XLM-R-large consistently outperformed other mPLMs. This is attributed to its training on the massive CommonCrawl dataset, providing a more comprehensive representation of the Korean language compared to Wikipedia-based models.
  • Effectiveness of Synthetic Data: The P-based strategy proved most effective, achieving a Matthew Correlation Coefficient (MCC) of 0.401 on the Google test set. However, the M-based strategy (MCC 0.364) remains a viable alternative for low-resource scenarios where parallel data is unavailable.
  • Cross-Engine Objectivity: A significant finding was the model's stability across different translation engines. Despite being trained on data from a single engine, the performance variance across Google, Amazon, and Microsoft was minimal. This indicates that the model successfully learned general characteristics of translation errors rather than engine-specific biases.

Conclusion

This research demonstrates that high-quality word-level QE models for Korean-English NMT can be developed using synthetic data generation strategies. By leveraging mPLMs like XLM-R, these models provide an objective means of assessing translation quality without the need for expensive human post-editing. Future work will focus on scaling the pseudo-QE datasets and improving the prediction of "Gap" tags to better identify missing words in translations.

Summary updated

Abstract

Quality estimation (QE) task aims to predict the machine translation (MT) quality well by referring to the source sentence and its MT output. The various applicability of QE proves the importance of QE research, but the enormous human labor to construct the QE dataset remains a challenge. This study proposes three automatic word-level pseudo-QE data construction strategies using a monolingual or parallel corpus and an external machine translator without human labor. We utilize these individual pseudo-QE datasets to finetune multilingual pretrained language models such as cross-lingual language models (XLM), XLM-RoBERTa, and multilingual BART and comparatively analyze the results. Considering the synthetic dataset creation setup, we attempt to validate the objectivity of the QE model by leveraging four test sets translated by external translators from Google, Amazon, Microsoft, and Systran. As a result, XLM-R-large shows the best performance among mPLMs. We also verify the reliability of the QE model through the close performance gaps between different test sets. To the best of our knowledge, this is the first study to experiment with word-level Korean-English QE.
BibTeX
@article{eo2022word,
  author = {Sugyeong Eo and Chanjun Park and Hyeonseok Moon and Jaehyung Seo and Heuiseok Lim},
  title = {{Word-Level Quality Estimation for Korean-English Neural Machine Translation}},
  journal = {IEEE Access},
  year = {2022},
  doi = {10.1109/access.2022.3169155},
}
Figure