Findings of the WMT 2018 Shared Task on Automatic Post-Editing

Rajen Chatterjee, Matteo Negri, Raphael Rubino, Marco TurchiEMNLP 2018

TL;DR

This paper presents the findings of the WMT 2018 Shared Task on Automatic Post-Editing (APE), where systems learn to automatically correct machine-translated text. The task highlighted that while APE systems significantly improve lower-quality phrase-based translations, they struggle more with already high-quality neural translations, suggesting that the quality of the initial translation is a more critical factor than the MT paradigm itself.

What Problem Does This Paper Address?

Machine translation (MT) systems, while increasingly powerful, still produce errors. Automatic Post-Editing (APE) aims to automatically correct these errors, reducing the effort required by human translators. The WMT shared task on APE has been running for several years to track progress in this field. This year's task focused on English-German translations in the Information Technology domain, specifically comparing APE performance on outputs from traditional phrase-based MT (PBSMT) and modern neural MT (NMT) systems. The core challenge is to develop APE systems that can effectively learn from human corrections and apply them to new translations, especially as MT quality itself improves.

Methods

The APE task involved participants developing systems that take a source sentence and its machine-translated target sentence as input and produce a corrected target sentence. Systems were trained on datasets of (source, target, human post-edit) triplets. The key methodological aspect is the use of neural network architectures, particularly the Transformer architecture, which has become standard for both MT and APE. Participants experimented with various configurations, including multi-source approaches (using both source and target sentences) and different training strategies. The evaluation relied on automatic metrics like TER (Translation Edit Rate) and BLEU, as well as human evaluation using a direct assessment (DA) approach.

Datasets

The task utilized English-German data from the Information Technology domain.

  • Training and Development Data: Consisted of (source, target, human post-edit) triplets.
  • PBSMT Subtask: Included data from WMT 2016 and 2017 APE tasks, totaling 28,000 instances.
  • NMT Subtask: Included 13,442 training and 1,000 development instances.
  • Test Data:
    • PBSMT Subtask: 2,000 newly released instances.
  • NMT Subtask: 1,023 instances.
  • Additional Training Data:
    • A corpus of 4.5 million artificially generated post-editing triplets (from the 2016 winning system).
  • The English-German section of the eSCAPE corpus, comprising 14.5 million instances generated via both phrase-based and neural translation.

Key Insights and Ideas

  • Translation Quality is King: The most significant finding is that the quality of the initial machine translation is a more crucial factor for APE success than the MT paradigm itself. APE systems achieved much larger improvements on lower-quality PBSMT outputs compared to higher-quality NMT outputs.
  • NMT is Harder to Improve: While neural APE models are powerful, they face greater challenges when correcting already good NMT outputs. This suggests that current APE techniques might be better suited for fixing gross errors rather than subtle improvements in high-quality translations.
  • Reordering is Less of an Issue for NMT: Neural translations generally have fewer reordering errors than phrase-based ones. This means APE systems working on NMT outputs need to focus more on other error types, like lexical choice, rather than reordering.
  • Synthetic Data Boosts Performance: The inclusion of large synthetic corpora significantly contributed to performance improvements, particularly in the PBSMT subtask.
  • APE Systems are Aggressive: For PBSMT, systems tend to modify a large percentage of sentences, but their precision (proportion of truly improved sentences) is still a key area for improvement.

Key Results

  • PBSMT Subtask: Top systems achieved substantial improvements over the baseline "do-nothing" system, with gains up to -6.24 TER and +9.53 BLEU points. This indicates a strong ability to model the post-editing style learned from training data.
  • NMT Subtask: Improvements were much smaller, with top results showing gains up to -0.38 TER and +0.8 BLEU points. This reflects the higher quality of the initial NMT translations and a smaller room for improvement.
  • Human Evaluation: Human evaluators generally ranked systems similarly to automatic metrics, confirming that APE systems significantly outperform the baseline MT output. However, even the best APE systems did not reach the quality of human post-edits.

Limitations and Future Work

  • NMT Subtask Difficulty: The paper acknowledges the higher difficulty of the NMT subtask due to the quality of translations and less training data. This makes it harder to draw firm conclusions and interpret results.
  • Over-correction Analysis: While the paper explores over-corrections (edits made to correct sentences), further analysis is needed to understand if these represent acceptable paraphrases or unwanted deviations from the desired style.
  • Lexical Choice Focus: For NMT, future work should focus on developing APE methods that can better handle lexical choice errors, as reordering is less of an issue.
  • Human Evaluation Scale: The paper suggests that more experiments with larger test sets and more evaluators are needed to validate certain findings from the human evaluation, particularly regarding the higher quality of NMT post-edits.

Summary updated

BibTeX
@inproceedings{chatterjee2018findings,
  author = {Rajen Chatterjee and Matteo Negri and Raphael Rubino and Marco Turchi},
  title = {{Findings of the WMT 2018 Shared Task on Automatic Post-Editing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure