TL;DR
This paper introduces a novel automatic post-editing (APE) system that uses statistical models to refine the output of a rule-based machine translation (RBMT) system. The APE system significantly improves translation quality and translator productivity, especially for more complex texts, and proposes a new, effortless human evaluation metric.
What Problem Does This Paper Address?
Machine translation (MT) systems, while improving, still produce raw translations that often require significant human effort to correct (post-editing). This is particularly true for rule-based MT (RBMT) systems, which, despite their linguistic grounding, struggle with adaptability and can produce awkward or incorrect translations. The challenge is to bridge the gap between the linguistic precision of RBMT and the adaptability of statistical MT (SMT) to achieve higher quality translations with increased translator productivity. This paper aims to address this by developing an APE system that leverages the strengths of both approaches.
Methods
The core of this work is an Automatic Post-Editing (APE) system built using statistical models. This APE system is trained to correct errors made by a commercial RBMT system. Essentially, it treats the RBMT output as the source text and the target language as the desired output, learning to "fix" the RBMT's mistakes. This approach combines the linguistic knowledge embedded in RBMT with the data-driven learning capabilities of SMT.
A key methodological contribution is a new human evaluation measure called "suitability." This binary metric assesses whether a translation is good enough to be manually post-edited with effort savings. It's designed to be quick and easy for human evaluators, focusing on whether they would prefer to edit the translation or start from scratch.
Datasets
The experiments were conducted using two distinct English-to-Spanish corpora:
- Parliament Corpus: Consists of documents from parliamentary proceedings, characterized by the peculiarities of oral language.
- Protocols Corpus: A collection of medical protocols, presenting a more challenging task due to factors like different companies involved in training and test sets, and out-of-domain data.
Key Insights and Ideas
- Synergy of RBMT and SMT: Combining RBMT with a statistical post-editing layer (APE) can yield better results than either system alone, especially for complex texts. The APE system inherits the robustness of RBMT while correcting its specific errors.
- APE's Complementary Role: The APE system doesn't just replace RBMT; it complements it, fixing errors without drastically altering the grammatical structure, making post-editing more efficient.
- Discrepancy in Evaluation: Automatic metrics (like BLEU and TER) and human evaluation can yield different conclusions about system performance. Human evaluators found APE to be significantly better than SMT for the Parliament corpus, even though SMT scored higher on automatic metrics.
- "Suitability" as a Practical Metric: The proposed "suitability" metric offers a more pragmatic assessment of post-editing effort and translator productivity than traditional automatic metrics.
Key Results
- Parliament Corpus: The APE system significantly improved translation quality, with 94% of its translations deemed "suitable" by human evaluators. This represents an average improvement of 59.5% over RBMT.
- Protocols Corpus: While less conclusive, the APE system still showed promising results, with an average improvement of 6.5% over RBMT and 67% of its translations deemed "suitable."
- Robustness: The RBMT system, and by extension the APE system built upon it, proved more robust to out-of-domain data compared to the standalone SMT system.
- Human vs. Automatic: For the Parliament corpus, human evaluation favored APE over SMT, despite SMT performing better on automatic metrics. For the Protocols corpus, human evaluators found the difference between APE and RBMT less significant than automatic metrics suggested.
Limitations and Future Work
- Single Reference: The use of a single reference translation for automatic evaluation is noted as potentially pessimistic.
- Effortless Evaluation: While the "suitability" metric is a step towards effortless human evaluation, integrating this process directly into a translator's workflow for on-the-fly evaluation is suggested as a future improvement.
- Error Identification: The paper identifies several sources of errors, which will guide future system enhancements.
- Automatic System Selection: Translators would benefit from a system that automatically selects the most suitable translation among different MT systems, a feature that could be developed further.
Summary updated