TL;DR
This paper introduces a novel multi-encoder Transformer network for Automatic Post-Editing (APE). By modifying the Transformer's attention mechanism to explicitly consider the relationship between the original machine translation output, the source sentence, and the desired post-edited output, the model achieves competitive results on the WMT 2018 APE shared task, particularly for PBSMT.
What Problem Does This Paper Address?
Machine translation (MT) systems, while improving, still produce errors. Automatic Post-Editing (APE) aims to correct these errors without retraining the MT system itself. The APE task can be framed as a translation problem from the MT output to the post-edited sentence, but crucially, the original source sentence also provides valuable context. This makes APE a multi-source translation problem, where the model needs to leverage information from both the MT output and the source sentence to produce a high-quality post-edited output. Previous approaches have explored various ways to combine these sources, but effectively modeling the specific error characteristics of MT output and its relationship to the source and target remains a challenge.
Methods
The core contribution of this paper is a multi-encoder Transformer network specifically adapted for APE. The standard Transformer architecture, known for its effectiveness in sequence-to-sequence tasks, is modified in two key ways:
- Dual Encoders: Instead of a single encoder for the input, the proposed model uses two separate encoders. One encoder processes the machine translation output (mt), and the other processes the original source sentence (src). Each encoder has its own self-attention and feed-forward layers.
- Modified Encoder-Decoder Attention: The crucial innovation lies in the decoder's attention mechanism. It incorporates two multi-head attention layers that attend to the outputs of the two encoders. One attention layer focuses on the relationship between the source sentence and the machine translation output (src→mt), while the other focuses on the relationship between the source sentence and the target post-edited sentence (src→pe). This allows the model to better understand which parts of the source sentence are relevant for correcting the MT output and for generating the final post-edited sentence. The authors hypothesize that this explicit modeling of the relationship between the original translation and the ideal translation is key to reducing the "gap" caused by MT errors.
Datasets
The experiments were conducted using data from the WMT 2018 official shared task on Automatic Post-Editing. This includes:
- PBSMT Task Data: Official training, development, and two test sets (2016, 2017).
- NMT Task Data: Official training and development sets.
- Artificial Training Data: The authors also utilized artificial training data (artificial-large and artificial-small) from previous WMT APE tasks to augment the official datasets.
Key Insights and Ideas
- APE as Gap Reduction, Not Just Translation: The paper argues that APE isn't just about translating from MT output to post-edited output. It's about actively reducing the systematic errors present in MT, which creates a "gap" between the MT output and the desired post-edited sentence. This perspective guides their architectural choices.
- Explicitly Modeling Source-MT and Source-Target Relations: By using dual encoders and a modified encoder-decoder attention, the model explicitly learns how the source sentence relates to both the flawed MT output and the correct post-edited output. This is a departure from simply concatenating or combining sources.
- Transformer's Adaptability: The Transformer architecture, with its powerful attention mechanism, proves to be a flexible foundation for adapting to the multi-source nature of APE.
Key Results
The proposed multi-encoder Transformer network achieved competitive results on the WMT 2018 APE shared task:
- PBSMT Task: The primary submission achieved a significant improvement over the baseline, with a reduction of -4.52 TER and an increase of +6.81 BLEU scores. This performance was comparable to the top results from the previous year's competition.
- NMT Task: For the NMT task, the improvement was more modest, with a reduction of -0.13 TER and an increase of +0.40 BLEU scores compared to the baseline. The authors suggest that the differing characteristics of the artificial data used for PBSMT might explain this difference.
- The paper also details various ensemble strategies (top-5, fix5, var5) that further refined the model's performance, as shown in Table 3.
Limitations and Future Work
- NMT Task Performance: The paper acknowledges that the improvement on the NMT task was less pronounced than on the PBSMT task. They hypothesize that this might be due to the nature of the artificial training data used, which may be more aligned with PBSMT characteristics.
- Future Work: While not explicitly detailed, the success on PBSMT suggests that further exploration of how to best leverage artificial data and potentially adapt the model to specific MT system error patterns could be fruitful. The authors' focus on the Transformer architecture also implies that further architectural refinements within the Transformer framework could yield additional gains.
Summary updated