MS-UEdin Submission to the WMT2018 APE Shared Task: Dual-Source Transformer for Automatic Post-Editing

Marcin Junczys-Dowmunt, Roman GrundkiewiczEMNLP 2018

TL;DR

This paper presents the MS-UEdin submission to the WMT2018 Automatic Post-Editing (APE) shared task, achieving state-of-the-art results on the SMT sub-task by leveraging dual-source transformer models and extensive parameter sharing. Interestingly, their NMT sub-task performance was surprisingly weak, leading them to question the utility of neural-on-neural APE.

What Problem Does This Paper Address?

Automatic Post-Editing (APE) aims to automatically correct machine-translated text to match human post-edited output. The challenge lies in creating systems that can effectively learn from limited post-editing data and generalize well. The WMT2018 APE shared task provided datasets for both Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) post-editing. The authors tackle the problem of improving APE systems, particularly by exploring novel architectures like dual-source transformers and effective parameter sharing strategies, while also investigating the performance of neural models on NMT output.

Methods

The core of the MS-UEdin submission is a dual-source transformer architecture. This builds upon the standard Transformer model by incorporating an additional encoder and a target-source multi-head attention component within each encoder block. This allows the model to attend to both the source text and the machine-translated (MT) text simultaneously.

Key methodological contributions include:

  • Parameter Sharing: They explored various forms of parameter sharing, including tying embeddings across encoders and decoder layers, and sharing all parameters between the two encoders. This is a crucial technique for improving performance when training data is scarce.
  • Dual-Source Transformer: This architecture is presented as a novel approach for APE, specifically designed to leverage both the source and MT inputs.
  • Data Augmentation: They utilized existing artificial data generated via round-trip translations and the eSCAPE corpus, augmenting the official training data to create a larger, more diverse training set.
  • Ensembling: For the NMT sub-task, they employed an ensemble of four models to boost performance.

Datasets

The experiments were conducted using the official WMT-2018 automatic post-editing data, which consists of triplets of (source text, MT output, human post-edited text). They also incorporated:

  • Artificial Data: Generated by the WMT2017 organizers via round-trip translations, filtered to match HTER statistics.
  • eSCAPE Corpus: A large, newly available corpus for APE, containing triplets generated from both SMT and NMT systems.

They experimented with different subsets of the eSCAPE data, including 1M, 2M, 4M, and the full ~8M sentence dataset.

Key Insights and Ideas

  • Parameter Sharing is Highly Beneficial for APE: Even when encoders are processing different languages (source vs. MT), sharing parameters significantly improved performance across multiple test sets. This suggests that the underlying linguistic structures learned by the encoders can be generalized.
  • Dual-Source Transformer Outperforms Previous Best: Their dual-source transformer, especially with parameter sharing, achieved state-of-the-art results on the SMT APE sub-task, surpassing previous best models by a considerable margin.
  • Neural-on-Neural APE Might Not Be Necessary (or Effective): The most surprising finding was the relatively poor performance of their neural APE system on NMT output, despite significant effort. This led them to hypothesize that neural-on-neural APE might not be as useful as initially thought, and that many improvements might stem from correcting basic errors (like BPE artifacts) in the MT output rather than complex linguistic correction.
  • Data Augmentation is Crucial: The combination of official, artificial, and eSCAPE data was essential for achieving strong results.

Key Results

  • SMT Sub-task: The MS-UEdin submission decisively won the SMT post-editing sub-task, establishing a new state-of-the-art. Their best model achieved a TER of 18.00 and BLEU of 72.52 on the test set, a significant improvement over the baseline.
  • NMT Sub-task: They were a very close second (or equal) in the NMT sub-task, with a TER of 16.50 compared to the winner's 16.46. However, they note that this was achieved with significantly less effort than their SMT system.
  • Parameter Sharing Impact: Tying embeddings and sharing encoders led to visible improvements in TER over their baseline. Sharing all encoder parameters further boosted performance.

Limitations and Future Work

  • Limited Training Time for NMT: The authors acknowledge that their NMT submission was trained for only about a day, and more training time might have yielded better results.
  • Questioning Neural-on-Neural APE: Their primary limitation is the observed weak performance on NMT output. They suggest that future tasks should focus on correcting general-domain on-line systems rather than neural-on-neural APE.
  • Availability of NMT Training Data: They propose that making the original NMT training data available would allow for a fairer comparison between pure NMT systems and APE systems.
  • Future Directions: They suggest exploring the feasibility of neural-on-neural automatic post-editing more thoroughly and investigating how to correct errors in general-domain on-line systems.

Summary updated

BibTeX
@inproceedings{junczysdowmunt2018ms,
  author = {Marcin Junczys-Dowmunt and Roman Grundkiewicz},
  title = {{MS-UEdin Submission to the WMT2018 APE Shared Task: Dual-Source Transformer for Automatic Post-Editing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure