TL;DR
This paper presents DFKI-MLT's systems for the WMT18 Automatic Post-Editing (APE) task, focusing on neural sequence-to-sequence models. They explored both recurrent and Transformer-based architectures, finding that the Transformer model achieved better performance, especially when post-editing phrase-based machine translation outputs.
What Problem Does This Paper Address?
Automatic Post-Editing (APE) aims to automatically correct errors in machine translated text, making it more human-readable and accurate. The WMT18 APE task introduced a significant challenge: not only to post-edit traditional phrase-based machine translation (PBMT) outputs but also to handle outputs from neural machine translation (NMT) systems. This requires APE models to be robust enough to deal with different types and patterns of errors introduced by these distinct MT paradigms. Furthermore, the availability of larger training datasets presented an opportunity to train more sophisticated APE models.
Methods
The DFKI-MLT group developed three APE systems, all based on neural sequence-to-sequence architectures trained using only target-language data. They employed two main neural network architectures:
- Recurrent Neural Network (RNN): This system used a 2-layered, unidirectional RNN with LSTM cells, incorporating global attention and input feeding. This is a standard approach for sequence-to-sequence tasks.
- Attention-Only (Transformer): This architecture, based on the "Attention Is All You Need" paper, utilizes multi-head attention and feed-forward layers without recurrence. They trained two versions of this model: "Transformer small" and "Transformer large," varying the number of attention heads and layer dimensions to explore their impact.
A key methodological innovation was the use of artificial tokens prepended to the source segments. These tokens indicated the type of MT system (PBMT or NMT) that generated the source text and the corpus it originated from. This allowed a single APE model to handle outputs from different MT systems and sources.
Datasets
The training data for the APE models was composed of:
- The official training sets provided for the WMT APE shared task since 2016.
- Two additional resources:
- Artificial training data (Junczys-Dowmunt and Grundkiewicz, 2016).
- The eSCAPE corpus (Negri et al., 2018).
These additional corpora were filtered using a bilingual cross-entropy difference approach. The authors also employed oversampling techniques to balance the amount of data from different sources and to equalize the proportion of PBMT and NMT data.
Key Insights and Ideas
- Unified APE for PBMT and NMT: The use of artificial tokens to signal the source of the MT output proved effective in enabling a single model to handle both PBMT and NMT outputs. This is a practical step towards more generalized APE systems.
- Transformer Architecture Dominance: The attention-only Transformer architecture significantly outperformed the recurrent RNN architecture for APE tasks, highlighting the power of self-attention mechanisms in capturing complex dependencies for post-editing.
- PBMT vs. NMT Post-Editing Difficulty: The systems showed a notable difference in performance between post-editing PBMT and NMT outputs. While improvements were seen for PBMT, post-editing NMT outputs led to degradation compared to the baseline.
Key Results
- PBMT Output: The "Transformer large" model significantly improved PBMT output according to BLEU scores, showing improvements of +0.4pt (post-edited MT), +0.39pt (reference), and +0.45pt (both). However, TER metrics did not show significant improvements over the baseline for PBMT.
- NMT Output: For NMT output experiments, all APE models showed degradation compared to the baseline across all evaluation metrics (BLEU and TER).
- Architecture Comparison: The attention-only Transformer models consistently outperformed the RNN model for both PBMT and NMT outputs.
Limitations and Future Work
- NMT Degradation: The degradation observed when post-editing NMT output is attributed to two main factors:
- A lower amount of available training data for NMT compared to PBMT.
- The baseline NMT output already achieved very high performance (indicated by BLEU scores), making it harder for the APE system to improve upon it.
- Vocabulary Coverage: While oversampling helped balance data amounts, it did not increase vocabulary coverage, which might be a limiting factor.
- Hyper-parameter Tuning: The authors acknowledge that their hyper-parameters were largely defaults and not extensively tuned. They plan to conduct further experiments with careful hyper-parameter tuning to potentially achieve better post-editing performances.
Summary updated