TL;DR
This paper introduces a novel approach to machine translation post-editing by framing it as a statistical phrase-based translation task. By training a phrase-based MT system to translate raw MT output into human-corrected text, they achieve significantly better quality than both the original rule-based MT and a standalone phrase-based MT system, demonstrating the power of combining these technologies.
What Problem Does This Paper Address?
Machine translation (MT) output, while improving, often still requires significant human correction (post-editing) to be usable. This manual process is labor-intensive and can be repetitive, as MT systems tend to make the same errors repeatedly. While rule-based MT systems can be customized, it's a costly and limited process. Statistical MT (SMT), particularly phrase-based MT (PBMT), offers a more adaptable solution by learning from data. However, even PBMT struggles to consistently produce human-readable translations. This paper tackles the challenge of automating the post-editing process to reduce human effort and improve translation quality by leveraging the strengths of both rule-based and statistical MT.
Methods
The core innovation is treating post-editing as a translation task. The authors use a phrase-based statistical machine translation system called Portage (developed at NRC Canada). Instead of translating from a source language to a target language, Portage is trained to translate from the raw output of a rule-based MT system (the "source") to the human-post-edited version of that output (the "target").
The Portage system itself is a standard PBMT architecture:
- Tokenization: Preprocessing of text.
- Decoding: A beam search algorithm to find the most probable translation hypotheses. This involves combining target-language phrases whose source-language counterparts match the input, drawing from large phrase tables.
- Rescoring: An additional log-linear model with more features (including IBM models) to reorder the hypotheses and select the best final output.
Crucially, the authors use the exact same Portage system and training procedure for both direct translation and for post-editing. They simply change the input data: for translation, it's the original source text; for post-editing, it's the raw MT output.
Datasets
The primary dataset used is from the Canadian government's Job Bank website. This corpus consists of parallel "blocks" of text, each containing:
- The original source language text (submitted by employers).
- The raw machine translation output (from a commercial rule-based MT system).
- The final, human-post-edited version (as published on the website).
The corpus contains less than one million words per language pair (English-to-French and French-to-English). For experiments, the data was split into training, validation, and test sets. A subset of 1000 blocks was used for training a secondary model (train-2), and the rest (train-1) was used for estimating IBM translation models and building phrase tables.
Key Insights and Ideas
- Post-editing as Translation: The fundamental insight is that the task of correcting MT errors is remarkably similar to translating from one language to another, making PBMT a natural fit.
- Synergy of Rule-Based and Statistical MT: Combining a rule-based MT system (for initial translation) with a statistical post-editing system (to refine it) yields superior results compared to using either system in isolation.
- Efficiency with Limited Data: The automatic post-editing (APE) approach shows significant improvements even with relatively small amounts of training data, suggesting its viability for specialized domains.
- APE Outperforms Standalone PBMT: Surprisingly, the APE system, even when trained on limited data, outperformed a standalone PBMT system trained on the same data. This highlights the effectiveness of the APE strategy in correcting specific types of errors.
Key Results
- Significant Quality Improvement: The phrase-based post-editing model improved the quality of translations, reducing the Translation Edit Rate (TER) by up to one-third and increasing BLEU scores by up to 50% (relative) compared to the baseline rule-based MT output.
- Outperforming Baselines: The APE system consistently outperformed both the original rule-based MT baseline and a standalone Portage PBMT translation system.
- Learning Curves: Learning curves showed that the APE system achieved strong performance with as few as 2000 blocks (approx. 25k words) for post-editing, and direct translation also showed rapid improvement with around 8000 blocks (approx. 100k words).
- Language Pair Differences: Performance varied between English-to-French and French-to-English, partly due to differences in the baseline system's performance and the amount of available training data.
Limitations and Future Work
- Potential for APE to be Outperformed: The authors acknowledge that while APE performs better with limited data, given sufficient training data, a direct PBMT system trained exclusively on the target domain might eventually outperform the APE approach. This is because APE is inherently limited by the errors of the front-end MT system.
- Future Work - Incorporating Source Input: A key research avenue is to modify the APE system to also consider the original source-language input, not just the raw MT output. This could involve introducing new features into the log-linear model that link target phrases to source text.
- Future Work - Combining Multiple MT Systems: The authors propose exploring a "combination MT system" where the APE system could potentially use outputs from multiple MT systems as secondary inputs, further enhancing robustness.
- Domain Specificity: The experiments highlight the domain-specific nature of both rule-based and statistical MT, emphasizing the need for appropriate training material for optimal performance.
Summary updated