TL;DR
This paper tackles the "over-correction" problem in automatic post-editing (APE) by introducing an "effort-aware" approach. By signaling to the APE system how much correction is likely needed, they aim to prevent unnecessary edits on already good machine translations, leading to improved performance, especially for high-quality MT outputs.
What Problem Does This Paper Address?
Machine translation (MT) systems have become remarkably good, but they still make errors. Automatic Post-Editing (APE) aims to fix these errors, essentially treating it as a sequence-to-sequence translation task where the MT output is the source and the desired corrected translation is the target. However, as MT quality improves, APE systems face a new challenge: "over-correction." This happens when the APE system, trying to be helpful, rephrases an already correct or near-correct MT output. While the rephrased sentence might still be a valid translation, it's penalized by standard evaluation metrics (like BLEU and TER) because it deviates from the human-annotated reference post-edit. This is particularly problematic when MT outputs are already of high quality.
Methods
The core innovation here is the introduction of an "effort-aware" mechanism. Inspired by multilingual NMT, the authors prepend a special token to both the source text and the MT output. This token signals the predicted amount of post-editing required. They propose three levels: "no post-edit," "light post-edit," and "heavy post-edit."
The APE system itself is built upon the Transformer architecture, a standard in neural MT. It uses a dual-encoder setup to process both the source text and the MT output. Crucially, they share parameters between the encoders and tie embeddings and decoder softmax layer weights, a common practice for efficiency and performance.
To determine the appropriate token during training, they use the Translation Edit Rate (TER) score between the MT output and the reference post-edit.
- No post-edit: For MT outputs with a TER of 0.
- Light post-edit: For MT outputs with a TER between 0 and 40.
- Heavy post-edit: For MT outputs with a TER greater than 40. (The 40 TER threshold is based on prior work suggesting human translators tend to rewrite from scratch above this point).
At test time, the oracle token (i.e., the true required edit level) isn't available. They explore two methods to predict this token:
- BERT: Fine-tuning a BERT model as a text classifier to predict the edit level.
- SIM: An information retrieval approach that finds similar training examples and uses their associated tokens.
To make the system robust to noisy predicted tokens, they also experiment with fine-tuning the APE system on in-domain data where a percentage of the tokens are randomly perturbed.
Datasets
The paper uses data from the WMT 2019 APE shared task. This includes:
- In-domain data: Real human post-edits.
- Synthetic data:
- eSCAPE dataset: Generated by treating target sentences as artificial human post-edits and machine-translated source sentences as MT output.
- A second synthetic dataset for the English-German subtask, created using round-trip translation from a German monolingual corpus.
Byte Pair Encoding (BPE) is applied to create a shared vocabulary across source text, MT output, and post-edit output.
Key Insights and Ideas
- The "over-correction" problem is real and detrimental: As MT gets better, APE systems need to be smarter than just blindly correcting.
- Signaling edit effort is beneficial: Prepending a token indicating the required post-editing effort can guide the APE system to be more conservative when appropriate, leading to better results.
- Human intuition can be encoded: The three-tier token system (no, light, heavy) mimics how a human might approach post-editing based on the perceived quality of the MT.
- Predicting the edit level is challenging: The performance drop when using predicted tokens (BERT or SIM) instead of oracle tokens highlights the difficulty of accurately assessing MT quality at test time.
- Robustness training is key: Fine-tuning with noisy tokens helps mitigate the negative impact of incorrect token predictions.
Key Results
- English-German subtask:
- With an "oracle" token (knowing the true edit level), APE systems improved MT output by up to +0.98 BLEU and -0.47 TER on development sets.
- On the test set, the best system without prepending a token achieved TER 15.08 and BLEU 76.76. Systems with predicted tokens showed a performance drop compared to the no-token baseline, but robust training (using noisy tokens) improved performance.
- English-Russian subtask:
- Due to the high quality of the MT output, none of the submitted systems were able to improve the MT output on the development or test sets. This underscores the difficulty of the over-correction problem in this scenario.
- However, even in this case, the system with the token showed slightly better performance than the one without, suggesting the token still provides some useful signal.
Limitations and Future Work
- Predicting the correct token at test time is a significant challenge: The performance drop when using predicted tokens highlights that accurately assessing MT quality in real-time remains an open problem.
- The En-Ru subtask showed limited gains: The high quality of MT outputs in this subtask meant that APE systems struggled to improve performance, indicating a need for further research into more reliable solutions for over-correction.
- Future work: The authors emphasize the need for further research into predicting MT quality more reliably, which would directly benefit their effort-aware APE approach. They also suggest that the number of tokens could be increased for finer-grained control over post-editing strategies.
Summary updated