TL;DR
This paper presents a three-step methodology to improve statistical machine translation (SMT) systems by leveraging noisy, free-form user edits from online translation services. By filtering, aligning, and then incorporating these edits, they demonstrate significant improvements in translation quality, even with limited user feedback data.
What Problem Does This Paper Address?
Statistical Machine Translation (SMT) systems, while widely used, still suffer from various translation errors (e.g., morphology, word order, lexical selection). These errors can be frustrating for users, especially in free online translation services. While user feedback is a valuable resource for improvement, it's often noisy and error-prone, especially when collected from non-professional users who may have limited language skills or provide partial edits. The challenge is to effectively filter and utilize this noisy feedback to improve SMT models without introducing more errors or requiring extensive expert annotation.
Methods
The authors propose a three-step methodology:
- Filtering User Edits: They developed a supervised classification approach to identify "useful" user edits (UEs) that genuinely improve the translation over the system's output. This involves training classifiers using features related to surface similarity, back-translation, noise, similarity-based measures, and quality estimation.
- Pivot-Based Alignment (SimTer): A novel alignment method called SimTer is introduced. It uses the original machine translation (TGT) as a pivot to align the source sentence (SRC) with the user-edited translation (UE). This process focuses on identifying and correcting specific errors made by the SMT system by projecting the TGT-UE alignment to SRC. SimTer leverages Translation Edit Rate (TER) for initial alignment and then uses similarity features to refine it, making it robust to noisy edits.
- Incorporating Aligned Phrase Pairs: The new, aligned phrase pairs extracted from the UEs are incorporated into the original SMT model. This is done through linear interpolation, where the weight of the UE-specific translation model is tuned to maximize translation quality (BLEU score) on a development set.
Datasets
- FAUST UE Corpus: A collection of 550 English-Spanish translation requests, system outputs, and user edits from the Reverso.net service. This corpus was used for training the user edit classifier.
- FAUST dev/test Corpus: A set of parallel data from Reverso.net with professional translator revisions, used for tuning and testing.
- Europarl v6 corpus: Used to build a base SMT system for tuning similarity function weights.
- EPPS corpus: Used for building base SMT systems and language models.
- WMT'12 Campaign Corpora: Including EPPS, News Commentary (NC), and United Nations (UN) parallel data, used for tuning interpolation parameters and validating the methodology.
- Common Crawl Corpus: A large, noisy collection of parallel texts automatically mined from the web, used to test the generalization ability of the methodology in a domain adaptation scenario.
Key Insights and Ideas
- Noisy feedback can be valuable: Even with imperfect user edits, a systematic approach can extract significant improvements for SMT systems.
- Filtering is crucial: Simply concatenating all user edits degrades performance. Identifying and selecting only the useful edits is paramount.
- Focusing on errors: The SimTer alignment method's strength lies in its ability to focus on the differences between the system output and the user edit, directly targeting translation errors.
- Generalization: The proposed methodology, particularly the filtering classifiers, shows robustness and generalizes well to different domains and data sources (e.g., user-edited data vs. web-crawled parallel corpora).
- Beyond lexical coverage: Improvements are not just due to adding new words (reducing OOV) but also to better lexical selection, reordering, and morphology.
Key Results
- The proposed three-step methodology significantly improves translation quality across various automatic metrics (BLEU, NIST, TER, METEOR) on both FAUST and Common Crawl datasets.
- In the FAUST scenario, the SimTer_0.6 & FFF+ strategy achieved significant improvements, with the best results obtained using only 50% of the user edits.
- Human evaluation on the FAUST test set showed that about 50-60% of translations changed, with a doubled number of improvements compared to quality decreases, particularly in lexical aspects and reordering.
- In the Common Crawl scenario, selecting just 25% of the data yielded the best improvement, demonstrating the effectiveness of the selection strategy even with noisy references.
- The SimTer alignment method was found to be more effective than mGIZA++ for this specific task, especially in the Common Crawl scenario, and was computationally efficient.
Limitations and Future Work
- Moderate classifier precision: The user edit classification achieved only moderate precision (around 70%), indicating the difficulty of perfectly identifying useful edits, even for humans. This suggests that the selection threshold is a trade-off between precision and recall.
- Potential for introducing errors: While improvements are significant, the enriched system can occasionally introduce incorrect words or expressions from wrongly selected or aligned examples, and may perform slightly worse in terms of adding/omitting spurious words.
- Future work: The authors suggest exploring joint "UE-selection" and "MT-enrichment" learning settings, although they note this would be computationally inefficient for their pipeline. They also mention that further analysis of alignment error rate is beyond the scope of the paper.
Summary updated