Quality Estimation and Translation Metrics via Pre-trained Word and Sentence Embeddings

Lisa Yankovskaya, Andre Tättar, Mark FishelACL 2019

TL;DR

This paper introduces two novel, lightweight neural network models for machine translation quality estimation (QE). By leveraging pre-trained multilingual embeddings (BERT and LASER), these models achieve competitive results in predicting translation quality (HTER and direct human assessment) without requiring extensive task-specific training data or human references.

What Problem Does This Paper Address?

Accurately assessing the quality of machine translation (MT) output is crucial, but traditional methods often rely on human references, which are expensive and time-consuming to obtain. Quality Estimation (QE) aims to predict translation quality without these references. A common proxy for QE is predicting the post-editing effort (HTER), which measures how many corrections a human translator would need to make. However, HTER doesn't always perfectly align with human judgment of overall quality. This paper tackles the challenge of developing effective QE models that are computationally efficient, require less data, and can predict both post-editing effort and direct human assessments.

Methods

The authors propose two neural network architectures for sentence-level QE. Both models are built upon pre-trained word and sentence embeddings, a departure from methods that train embeddings from scratch.

  • LABE (Language-Agnostic BERT Embeddings): This model uses embeddings extracted from two powerful pre-trained multilingual toolkits: BERT (for language understanding) and LASER (for sentence embeddings across many languages). These embeddings are fed into a feed-forward neural network with a sigmoid output layer to predict the quality score.
  • LABEL (Language-Agnostic BERT Embeddings + Log Probability): This is an extension of LABE. In addition to BERT and LASER embeddings, it incorporates the log probability score obtained from a trained neural MT system (specifically, a Transformer model trained using the Sockeye toolkit). This additional feature aims to provide the model with more information about the translation's likelihood.

Both models are implemented using a four-layered feed-forward neural network with a sigmoid activation function, making them relatively lightweight.

Datasets

The paper utilizes several datasets for training and evaluation:

  • WMT16-WMT18 Shared Tasks: Data from these tasks on sentence-level QE for English-German (En-De), German-English (De-En), and English-Czech (En-Cs) were used. These datasets contain translations from statistical and neural MT systems, along with HTER scores.
  • Extended Datasets: The authors also created extended datasets from previous years' WMT data to ensure more reliable scores and mitigate skewed distributions.
  • News Translation Tasks (2015-2018): Data from these tasks were used for direct human assessment (DA) prediction. This includes language pairs like English-Finnish (En-Fi) and English-Russian (En-Ru). The DA data consists of source sentences and their MT outputs, but crucially, without reference translations.
  • DAseg-newstest2016, DAseg-newstest2015, DAseg-newstest2017: These datasets were used as test and fine-tuning sets for DA prediction.

Key Insights and Ideas

  • Leveraging Pre-trained Embeddings is Powerful: The core idea that pre-trained, freely available embeddings (BERT, LASER) can be highly effective for QE without extensive task-specific pre-training is a significant contribution. This drastically reduces computational cost and data requirements.
  • HTER vs. Direct Assessment Nuances: The paper highlights that HTER (post-editing effort) and direct human assessment (DA) are not perfectly correlated. A translation might have minor errors that lead to a high HTER score but still be perceived as high quality by humans. This motivates the need for QE models that can predict both.
  • Lightweight Models Can Compete: The proposed neural architectures are relatively simple feed-forward networks, yet they achieve performance comparable to or better than more complex methods, especially when compared to the baseline "deepQuest" model.
  • Multi-language Embeddings for Cross-lingual Transfer: The use of multilingual embeddings like LASER and BERT inherently allows for potential cross-lingual transfer, which is explored by training models on multiple language pairs.

Key Results

  • LABEL Outperforms LABE: The LABEL model, which includes log probability from an MT system, generally achieved slightly better Pearson and Spearman correlation coefficients for HTER prediction across various language pairs compared to the LABE model.
  • Comparable to Metrics: The proposed models achieved comparable results to traditional MT metrics (like BLEU, TER) on HTER prediction, even though the QE models do not use reference sentences, which metrics typically require.
  • Performance on WMT19: Results on the smaller WMT19 dataset were less impressive than on extended datasets, suggesting potential issues with data distribution or task setup.
  • DA Prediction: For direct human assessment prediction, the models showed promising results. Training with a single language pair often yielded the best performance, though multi-language models also performed well. Fine-tuning on human assessment data further improved results.

Limitations and Future Work

  • WMT19 Dataset Performance: The authors note that the performance on the small WMT19 dataset was not as strong, and they speculate this might be due to skewed data distribution.
  • Data Scarcity for DA: Obtaining large datasets for direct human assessment is challenging, which limits the scope of DA prediction experiments.
  • Future Work: The paper suggests exploring the use of other pre-trained embeddings and investigating the impact of different MT systems for log probability features. Further analysis of the differences between HTER and DA prediction and how to best model them is also implied.

Summary updated

BibTeX
@inproceedings{yankovskaya2019quality,
  author = {Lisa Yankovskaya and Andre Tättar and Mark Fishel},
  title = {{Quality Estimation and Translation Metrics via Pre-trained Word and Sentence Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure