TL;DR
This paper introduces a novel framework for actively learning to build multiple statistical machine translation (SMT) systems simultaneously, allowing for the efficient addition of a new language to an existing multilingual parallel corpus. By leveraging co-training and consensus translations across multiple MT systems, they achieve significant improvements in translation quality compared to random sentence selection, even with limited initial human-labeled data.
What Problem Does This Paper Address?
Training high-quality statistical machine translation (SMT) systems requires vast amounts of parallel text (sentences aligned across two languages). While large multilingual parallel corpora exist (like the European Parliament proceedings), adding a new language to such a collection and building translation systems for it from scratch is a data-intensive and costly process. The challenge lies in efficiently selecting the most informative sentences for human translation to maximize the learning gain for multiple MT systems simultaneously, rather than just for a single language pair. This is particularly difficult when starting with a small amount of labeled data for the new target language, as initial translation quality is poor and requires a substantial injection of human effort to become effective.
Methods
The core of this work is an active learning (AL) framework designed for multilingual settings. Instead of training a single MT system, they train multiple MT systems from each source language in the existing multilingual corpus to the new target language.
Key methodological contributions include:
- Multilingual AL Framework: The AL process iteratively selects sentences from a large pool of unlabeled data to be human-translated. This selection is guided by the collective "difficulty" of translating these sentences across all the MT systems being trained.
- Co-training with Consensus Translations: When training multiple MT systems, they leverage the idea that different systems offer distinct "views" on the translation. They employ a co-training strategy where systems are retrained using not only human-labeled data but also "pseudo-labeled" data generated by other systems in the ensemble. A crucial component is using consensus translations – a single high-quality translation derived from the n-best lists of multiple MT systems – to facilitate this co-training. This allows systems to learn from each other's strengths.
- Novel Sentence Selection Strategies: The paper proposes several sophisticated sentence selection methods beyond simple random selection. These include:
- Alternating Selection: Iteratively focusing on improving one MT system at a time.
- Combined Ranking: Combining the rankings of sentences based on their importance for each individual translation task (source language to target language).
- Disagreement-Based Selection: Identifying sentences where the candidate translations from different MT systems show significant disagreement, indicating potential difficulty and thus high learning value.
- Phrase-Based SMT Integration: The methods are designed to work with phrase-based SMT models, focusing on selecting sentences that offer valuable phrases (both regular and "out-of-vocabulary" or OOV phrases) for improving the phrase table and overall model generalization.
Datasets
The primary dataset used is the EuroParl corpus, a large multilingual parallel corpus. For their experiments, they created a multilingual parallel corpus of 653,513 sentences, excluding a specific portion used for testing. They subsampled 5,000 sentences as the initial labeled data (L) and 20,000 sentences as the pool of unlabeled data (U). The test set consisted of 2,000 multi-language sentences from the Q4/2000 portion of the EuroParl corpus.
Key Insights and Ideas
- Multilingualism as a Learning Advantage: The multilingual setting isn't just about parallel data; it's an opportunity for AL. Different source languages can provide complementary information, allowing for more efficient learning than single-language-pair AL.
- Co-training Boosts Performance: Using consensus translations to enable co-training between multiple MT systems significantly improves performance compared to self-training (where a system only learns from its own outputs). This highlights the power of leveraging diverse perspectives.
- Disagreement Signals Difficulty: When multiple MT systems struggle to agree on a translation for a sentence, it's a strong indicator that the sentence is challenging and thus highly valuable for human annotation. This is a more effective signal than random selection.
- Phrase-Level Information is Key: The proposed sentence selection methods effectively leverage information about regular and OOV phrases, recognizing that improving the phrase table is crucial for phrase-based SMT.
Key Results
- Significant Improvement over Random Baseline: The proposed active learning methods, particularly the "Combined Rank" strategy, consistently outperformed the strong random sentence selection baseline across various multilingual experiments.
- Co-training Outperforms Self-Training: In multilingual settings, co-training achieved approximately 1 BLEU point improvement over self-training.
- Single-Language Pair Performance: The proposed sentence selection methods also outperformed existing methods like "GeomPhrase" in single-language pair settings.
- Quantitative Gains: While specific BLEU scores vary across experiments, the figures and tables demonstrate consistent gains in translation quality (measured by BLEU, WER, and PER) as more sentences are added through active learning, compared to random selection. For instance, in Germanic languages to English experiments, co-training with "Combined Rank" showed strong performance.
Limitations and Future Work
- True vs. Estimated Phrase Distributions: The paper notes that while their estimated phrase distributions capture some information, they don't perfectly match the true distributions and don't fully capture the observed Zipfian (power law) behavior of phrases. Enhancing these estimations is a direction for future work.
- Disagreement Method Performance: The disagreement-based selection methods underperformed the baseline in some Germanic language experiments, suggesting that their effectiveness might be context-dependent.
- Future Directions: The authors suggest exploring triangulation techniques to further enhance the ensemble of MT systems and plan to incorporate this into their active learning approach. They also acknowledge that future work should focus on omitting noisy data.
Summary updated