RTM results for Predicting Translation Performance

Ergun BiçiciEMNLP 2018

TL;DR

This paper presents an advancement in predicting machine translation quality without human references, achieving top results in the WMT18 Quality Estimation task. By refining their Referential Translation Machine (RTM) models with improved prediction combination and deeper learning architectures, the authors demonstrate a language-independent approach that significantly boosts performance, particularly for English-to-German NMT.

What Problem Does This Paper Address?

The core challenge this paper tackles is Quality Estimation (QE) for machine translation (MT). Specifically, it focuses on predicting how good a translation is without having access to the original, human-created "gold standard" translation. This is crucial because obtaining human references is expensive and time-consuming, making real-time quality assessment difficult. The paper addresses the WMT18 Quality Estimation Task (QET18), which involves predicting translation quality at various levels: sentence, word, phrase, and document. The goal is to develop models that can accurately estimate translation quality across different language pairs and MT paradigms (SMT and NMT) without requiring task-specific information.

Methods

The authors build upon their existing Referential Translation Machine (RTM) framework. The key innovations and techniques employed are:

  • Improved Prediction Combination: RTMs use "interpretants" – data points close to the task instances – to understand the context and difficulty of translation. This paper enhances how predictions from multiple RTM models are combined. They explore:
  • Stacking: Using predictions from base models as features for a second-level predictor (Figure 3).
  • Mixture of Experts (MoE): Averaging predictions from the top k models, weighted by their performance on the training set (Equation (1) and (2)). This aims to create a combined prediction that is better than any single component.
  • Deeper Prediction Models: They investigate using multilayer perceptrons (MLPs) to build more complex prediction models, moving beyond simpler regression techniques.
  • Feature Engineering: RTMs generate features by measuring the "closeness" of test instances to training instances using interpretants. The paper mentions that the number of features can vary based on n-gram orders.
  • Learning Models: A variety of standard machine learning models are used for prediction, including ridge regression, k-nearest neighbors, support vector regression, AdaBoost, gradient tree boosting, extremely randomized trees, and MLPs, often in conjunction with feature selection (FS) and partial least squares (PLS).

Datasets

The experiments are conducted on datasets from the WMT18 Quality Estimation Task (QET18). These datasets cover various language pairs and MT systems:

  • English-German (EN-DE): Both phrase-based SMT and NMT.
  • German-English (DE-EN): SMT.
  • English-Latvian (EN-LV): Both SMT and NMT.
  • English-Czech (EN-CS): SMT.
  • English-French (EN-FR): SMT.

The tasks involve predicting:

  • Task 1: HTER (human-targeted translation edit rate) scores (sentence-level).
  • Task 2: Binary classification of words.
  • Task 3: Binary classification of phrases.
  • Task 4: Multi-dimensional quality metrics (MQM) (document-level).

Table 1 provides a breakdown of the number of instances and interpretants used for each task.

Key Insights and Ideas

  • RTMs are powerful and language-independent: The core strength of RTMs is their ability to perform well across different language pairs and MT systems without needing specific linguistic knowledge or resources for each task. This makes them highly adaptable.
  • Combining predictions is key: Simply using a single RTM model is less effective than combining predictions from multiple models. Both stacking and mixture of experts approaches show promise in improving overall performance.
  • Deeper models can capture more complexity: The exploration of MLPs suggests that more complex architectures can potentially learn more nuanced relationships in the data, leading to better quality estimation.
  • Document-level estimation is challenging but achievable: While sentence-level tasks often yield better results, the paper shows that document-level RTM models can also achieve competitive performance, especially when using specific modeling techniques for documents.

Key Results

The paper highlights several significant achievements:

  • Top performance in WMT18: The RTM models achieved the 3rd best system overall in the WMT18 QET task.
  • Lowest RMSE for EN-DE NMT: They achieved the lowest RMSE for the English-to-German NMT sentence-level prediction task.
  • Competitive document-level results: For the document-level task (Task 4), their RTM models achieved competitive results, with one approach denoted as "doc" outperforming "sent" and obtaining top results in QET16.
  • Specific task successes: RTMs became the 1st system for English-to-German NMT quality prediction in QET18. They also achieved the lowest RMSE in EN-DE NMT.
  • General ranking: RTMs achieved the 1st system in EN-DE NMT and became the 3rd system in general for sentence-level prediction.

Limitations and Future Work

  • Test set references not released: The authors note that the references for the test sets were not released at the time of writing, which might limit direct comparison with some external benchmarks.
  • Further exploration of MoE: While the Mixture of Experts approach showed promise, the paper mentions filtering out results with relative evaluation metrics higher than 1. Further research could explore more sophisticated ways to combine expert predictions.
  • Document-level modeling: The paper compares two document-level modeling techniques. Future work could delve deeper into optimizing these approaches or exploring entirely new methods for document-level quality estimation.
  • Language independence: While RTMs are inherently language-independent, exploring their performance on a wider range of low-resource languages could be a valuable future direction.

Summary updated

BibTeX
@inproceedings{bicici2018rtm,
  author = {Ergun Biçici},
  title = {{RTM results for Predicting Translation Performance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure