Keep It or Not: Word Level Quality Estimation for Post-Editing

Prasenjit Basu, Santanu Pal, Sudip Kumar NaskarEMNLP 2018

TL;DR

This paper tackles the challenge of predicting whether individual words in a machine translation (MT) output should be kept or modified during post-editing. They propose a novel approach using word embeddings (Doc2Vec) to measure semantic similarity between source and target word contexts, outperforming traditional Bag-of-Words methods. This work is crucial for improving the efficiency and quality of human post-editing workflows.

What Problem Does This Paper Address?

Machine translation (MT) systems have become increasingly sophisticated, but their output still requires human review and correction by post-editors. A significant portion of post-editing effort involves deciding whether each word in the MT output is correct and should be retained, or if it needs to be modified or deleted. This word-level decision-making is time-consuming and can be subjective. The paper addresses the problem of automating this word-level quality estimation (QE) task. Specifically, it aims to predict, for each word in an MT output, whether it is a correct translation of its source counterpart and should be kept ("OK") or if it's an error and needs correction ("BAD"). This prediction is made without direct comparison to a human-generated reference translation, making it a challenging but valuable task for streamlining post-editing.

Methods

The core of the proposed approach is to measure the semantic similarity between the source context of a target MT word and the context in which that word is retained in a human post-edited (PE) translation. This similarity score is then used to predict whether the MT word is likely to be correct.

The authors explore two main methods for capturing this similarity:

  • Bag-of-Words (BoW) Model: This traditional approach represents the source context and the retained PE context as "bags" of words. The similarity between these bags is then calculated using cosine similarity.
  • Document-to-Vector (Doc2Vec) Model: This is the novel contribution. Instead of treating contexts as simple bags of words, Doc2Vec (specifically, the Paragraph Vector model) represents both the source context and the retained PE context as fixed-length vectors. These vectors capture deeper semantic relationships. The similarity between these document vectors is then computed.

For both models, the authors use the Translation Error Rate (TER) alignment on training data to identify which MT words were retained in the PE. They then establish a threshold on the similarity scores (derived from either BoW or Doc2Vec) to classify MT words as "OK" or "BAD." The Kneedle algorithm is employed to find the optimal threshold on the development set by analyzing the relationship between similarity score segments and the F1-mult metric (a combined F1 score for both OK and BAD classes).

Datasets

The experiments were conducted using the WMT-2018 English–German (EN–DE) word level Quality Estimation (QE task2) dataset. This dataset consists of parallel source-target sentences, MT outputs, and corresponding human post-editions. The paper notes that stop words were removed from the training and test data for both English and German to prevent them from skewing the similarity calculations.

Key Insights and Ideas

  • Semantic Similarity is Key: The fundamental idea is that if the source context and the context where the MT word is retained in post-editing are semantically similar, the MT word is likely correct.
  • Doc2Vec Captures Deeper Semantics: The Doc2Vec model, by representing contexts as dense vectors, is shown to capture semantic nuances that the simpler Bag-of-Words model misses. This leads to better performance in predicting word-level quality.
  • Imbalanced Data is a Hurdle: The datasets are highly imbalanced, with a large majority of MT words being retained (OK). This makes it challenging to achieve high performance, as even a simple baseline can achieve a decent F1 score by predicting "OK" for most words.
  • Thresholding is Crucial: Finding the right threshold for similarity scores is critical for accurate classification. The Kneedle algorithm proves effective in identifying these optimal thresholds.

Key Results

The experimental results indicate that the Doc2Vec-based model significantly outperforms the Bag-of-Words model for the word-level QE task. Specifically, the Doc2Vec model achieved better F1 scores for both the "OK" and "BAD" classes on the WMT18 test set. While specific F1-mult scores are presented in Table 5, the key takeaway is the superiority of the semantic embedding approach. The paper also notes that their model can only predict QE for words seen in the training data; unseen words (including stop words) are automatically retained.

Limitations and Future Work

  • Limited to Seen Words: The current model can only predict quality for words that appeared in the training data. Words not encountered during training are automatically classified as "OK."
  • Stop Word Removal Impact: The authors acknowledge that removing stop words from training and test data might have led to "not-up-to-the-mark performance" for the BoW model.
  • Dataset Size: The authors suggest that larger datasets could further improve the performance of Doc2Vec models, as they tend to benefit from more extensive training data.
  • Future Directions: The paper proposes extending the work to phrase-level QE and to identifying missing words or source words that directly lead to errors.

Summary updated

BibTeX
@inproceedings{basu2018keep,
  author = {Prasenjit Basu and Santanu Pal and Sudip Kumar Naskar},
  title = {{Keep It or Not: Word Level Quality Estimation for Post-Editing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure