Multi-source transformer with combined losses for automatic post editing

Amirhossein Tebbifakhr, Ruchit Agrawal, Matteo Negri, Marco TurchiEMNLP 2018

TL;DR

This paper introduces a novel multi-source Transformer model for Automatic Post-Editing (APE), a task focused on correcting machine translation outputs. By leveraging both the source and the raw translation, and employing a combination of sequence-level loss functions, their model achieves state-of-the-art results on the WMT 2018 APE shared task, outperforming previous methods while being simpler and more efficient.

What Problem Does This Paper Address?

Automatic Post-Editing (APE) aims to automatically correct errors in machine translation (MT) outputs, making them more human-readable and usable. This is particularly important because users often rely on "black-box" MT systems (like Google Translate) whose internal workings cannot be modified. APE offers a way to improve translation quality without altering the original MT engine. While recent approaches have shown success by using multi-source models that consider both the original source sentence and the MT output, these models can be complex and computationally expensive, hindering their adoption in real-world translation industries. This paper addresses the need for a more efficient and maintainable APE system that still achieves high performance.

Methods

The core of this work is the adaptation of the Transformer architecture for multi-source APE. Unlike standard sequence-to-sequence models, their approach uses two encoders: one for the source sentence and one for the MT output to be corrected. The outputs of these encoders are then concatenated and used as the "key" in the attention mechanism, allowing the decoder to effectively attend to both sources.

A key innovation is the use of combined sequence-level loss functions. Instead of relying solely on token-level Maximum Likelihood Estimation (MLE), which suffers from "exposure bias" (where the model only sees ground truth during training but has to predict based on its own outputs during inference), they incorporate Minimum Risk Training (MRT). MRT aims to directly optimize for sequence-level evaluation metrics like TER and BLEU. They explore different combinations of MLE and MRT, finding that a linear combination of both, where the reference is considered in the MRT loss, yields the best results.

Additionally, they employ ad hoc pre-processing for German language data, including a modified snowball stemming, empirical compound splitting, and Byte Pair Encoding (BPE). For English source sentences, only BPE is used for vocabulary reduction.

Datasets

The paper utilizes both in-domain and synthetic datasets for training and evaluation.

  • In-domain Data:
    • PBSMT subtask: 28K English-German triplets (source, MT output from a phrase-based system, human post-edit).
  • NMT subtask: 13K English-German triplets (source, MT output from a neural system, human post-edit).
  • 1K triplets from each set were reserved for development.
  • Synthetic Data:
    • eSCAPE corpus: 7.2M English-German triplets, generated using a round-trip translation protocol.
  • Artificial corpus: 4.0M English-German triplets from Junczys-Dowmunt and Grundkiewicz (2016).

The eSCAPE data underwent cleaning steps to remove triplets with unusual length ratios or non-English source sentences/non-German post-edits.

Key Insights and Ideas

  • Multi-source Transformer for APE: Adapting the Transformer, known for its success in MT, to a multi-source APE setting proves effective, allowing the model to leverage richer contextual information.
  • Sequence-Level Loss is Crucial: Moving beyond token-level MLE to sequence-level losses like MRT is essential for directly optimizing the metrics used for evaluating APE performance (TER and BLEU).
  • Combined Loss Functions Offer the Best of Both Worlds: A simple linear combination of MLE and MRT, particularly when the reference is included in the MRT calculation, allows the model to learn from both ground truth and the overall sequence quality, mitigating exposure bias and improving performance.
  • Simplicity Trumps Complexity for Industrial Use: The authors emphasize that their end-to-end, single-network approach is more practical for industry than complex ensemble or re-ranking methods, despite achieving comparable or better results.

Key Results

The proposed multi-source Transformer with combined losses achieved strong results on the WMT 2018 APE shared task:

  • PBSMT Subtask: Their top submission ranked second out of 11, achieving improvements of up to -5.3 TER and +8.23 BLEU points over the baseline.
  • NMT Subtask: Their top submission ranked first out of 10. While gains were smaller due to the higher quality of NMT inputs, they still achieved improvements of up to -0.38 TER and +0.8 BLEU.
  • On the development set, their best model (MRT+MLE) outperformed the best system from the previous year's shared task by up to -1.27 TER and +1.23 BLEU.

Limitations and Future Work

  • German Pre-processing: The paper mentions an "ad hoc" pre-processing pipeline for German. While effective, it might not be universally applicable or optimal for other languages.
  • Optimal Loss Combination: The authors note that the optimal weighting factor ($\alpha$) for the combined loss function (MRT + MLE) was set to 0.5 for equal importance and that empirical estimation of optimal values is left for future work.
  • NMT Gains: The smaller improvements on the NMT subtask are attributed to the already high quality of the input translations, leaving less room for error correction. Future work could explore how to further push the boundaries on these high-quality inputs.
  • Industrial Deployment: While the paper advocates for simplicity, further research could explore the trade-offs between model complexity and performance for specific industrial constraints.

Summary updated

BibTeX
@inproceedings{tebbifakhr2018multi,
  author = {Amirhossein Tebbifakhr and Ruchit Agrawal and Matteo Negri and Marco Turchi},
  title = {{Multi-source transformer with combined losses for automatic post editing}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure