TL;DR
This paper introduces a novel Transformer-based model for Automatic Post-Editing (APE) that enhances the encoder to jointly represent both the machine-translated text and its source, and equips the decoder with multi-source attention mechanisms. This approach significantly improves post-editing performance, achieving the second-best result in the WMT19 APE shared task.
What Problem Does This Paper Address?
Automatic Post-Editing (APE) aims to automatically correct errors in machine translation (MT) outputs to produce higher-quality translations. While neural APE models have shown promise, existing approaches often treat the MT output and its source sentence independently. This paper argues that the errors in MT are intrinsically linked to the source, and that a more integrated approach is needed. Specifically, they address the challenge of effectively leveraging both the source sentence (assumed to be error-free) and the MT output within a unified model to generate a better post-edited sentence.
Methods
The core of the proposed model is an extension of the Transformer architecture.
- Joint Encoder: Instead of separate encoders for the source (src) and machine translation (mt) inputs, this model uses a single encoder that generates a "joint representation." This is achieved by incorporating the source context into the MT's hidden states through residual connections and multi-head attention. This results in both an encoded source representation ($E_{src}$) and a joint representation ($E_{joint}$).
- Multi-Source Attention Decoder: The decoder is modified to incorporate two types of multi-source attention layers. These layers allow the decoder to attend to both the $E_{src}$ and $E_{joint}$ representations from the encoder.
- Parallel Attention: The decoder's hidden state attends to both $E_{src}$ and $E_{joint}$ simultaneously, with the outputs then linearly combined.
- Sequential Attention: The decoder's hidden state first attends to $E_{joint}$, and then the result attends to $E_{src}$ (or vice-versa, depending on the specific implementation).
- Teacher-Forcing Ratio Adjustment: To mitigate "exposure bias" (the discrepancy between training with ground truth and inference with generated outputs), the authors employ a two-step training process. In the second phase, they adjust the teacher-forcing ratio, stochastically feeding either the ground truth or the model's previous prediction as input to the decoder.
- Ensemble: Multiple model variants, differing in attention types and masking options, are trained and then combined using an ensemble approach to further boost performance.
Datasets
- WMT19 Official English-German APE Dataset: Used for training and development.
- eSCAPE NMT Dataset: Adopted as additional training data. This dataset was filtered based on sentence length, TER score, and reciprocal length ratio to match the characteristics of the official training data.
Key Insights and Ideas
- Synergy between Source and MT: The paper strongly advocates for and demonstrates the benefit of jointly encoding the source and MT output. The intuition is that MT errors are not random but are a function of the source, and thus, a model that understands this relationship should perform better.
- Multi-Source Attention is Crucial: The introduction of multi-source attention in the decoder, allowing it to query both the encoded source and the joint representation, is a key innovation. This enables the decoder to selectively draw information from both sources as needed.
- Teacher-Forcing Tuning Matters: Empirically adjusting the teacher-forcing ratio during fine-tuning proved effective in improving post-editing performance, highlighting the importance of bridging the gap between training and inference.
- Ensembling for Robustness: Combining multiple model variants, even those with slight architectural differences, led to further performance gains, suggesting that different models capture different aspects of the post-editing task.
Key Results
The proposed model achieved significant improvements over the baseline and previous state-of-the-art results.
- Primary Submission: Achieved -0.73 in TER and +1.49 in BLEU compared to the baseline.
- Ranking: The primary submission ranked second out of 18 systems in the WMT19 APE shared task, while also achieving the highest BLEU score among all submissions.
- Ensemble Performance: Ensemble models showed slightly worse TER scores but better BLEU scores than the best fine-tuned variants, demonstrating the benefit of combining diverse models.
Limitations and Future Work
- Masking Option: The paper explored an optional "future mask" in the encoder's self-attention layer, mimicking the sequential nature of MT decoding. Experiments were conducted for both cases (with and without this mask), but the paper doesn't explicitly state which performed better or if this was a significant factor in the final submission.
- Ensemble Strategy: While ensemble methods were used, the paper mentions selecting "top-N candidates" for ensemble, implying that the specific choice of N and the exact ensemble strategy could be further explored.
- Future Work: The authors suggest that their model architecture and training strategies can be applied to other sequence-to-sequence tasks. The paper also implicitly points towards further exploration of different multi-source attention mechanisms and ensemble strategies.
Summary updated