TL;DR
This paper introduces a novel approach to machine translation (MT) by leveraging "variant corpora" – parallel text datasets with the same meaning but different linguistic expressions. By training separate MT models on these variants and intelligently combining their outputs, they achieve significantly better translation quality than traditional methods that merge all data, especially by reducing ambiguities.
What Problem Does This Paper Address?
Traditional corpus-based machine translation (MT) systems often struggle with sentences that present linguistic phenomena not well-represented in their training data. While simply increasing the size of the training corpus by merging diverse sources might seem like a solution, it can paradoxically lead to more translation ambiguities and errors. The authors aim to address this by proposing a method that exploits the strengths of specialized MT models, each trained on a specific "variant" of a corpus, to improve overall translation accuracy and robustness.
Methods
The core of the proposed method involves:
- Variant Corpus Training: Instead of merging all available parallel text data, the authors train individual MT "engines" (models) on distinct "variant corpora." These variant corpora contain the same content but differ in vocabulary, syntax, or orthography. This specialization allows each engine to focus on specific linguistic phenomena.
- Hypothesis Selection: For a given input sentence, each specialized MT engine generates multiple translation hypotheses. A statistical method, using language and translation models, is employed to select the best hypothesis from each engine. This selection process uses statistical tests (like Kruskal-Wallis) to determine if one hypothesis is significantly better than others.
- Variant Selection: After selecting the best hypothesis from each variant-trained engine, a final selection is made. This involves comparing the statistical significance scores of the hypotheses from different variants. If one variant shows a significant advantage for a particular sentence, its hypothesis is chosen. If no significant difference is found, or if differences exist across multiple variants, the hypothesis from the engine that performed best on a development set is selected.
- Multi-Engine Extension: The approach is extended to multi-engine MT, where multiple MT engines are trained on each variant. This further diversifies the translation hypotheses available for selection.
Datasets
The experiments were conducted on the Chinese-English (CE) translation task from the travel domain, using data from the IWSLT 2005 evaluation campaign.
- BTECO: The original Japanese-English Basic Travel Expressions Corpus (BTEC), which was translated into Chinese, resulting in a large corpus of approximately 500K sentence pairs.
- BTECV: A variant corpus created by translating parts of the original English corpus directly into Chinese, resulting in a smaller corpus of about 162K sentence pairs.
The paper notes that only a small percentage of sentences and vocabulary overlap between BTECO and BTECV, highlighting their distinct nature.
Key Insights and Ideas
- Specialization Trumps Generalization (Sometimes): Training MT models on specialized "variant" corpora, rather than a single large, merged corpus, can lead to better translation quality by reducing ambiguity.
- Exploiting Linguistic Nuances: Different variants capture different linguistic phenomena (literalness, syntax, lexical choice, orthography). By training separate models, the system can leverage these specific strengths.
- Intelligent Combination is Key: Simply merging data is less effective than a sophisticated method for selecting the best translation hypothesis from specialized models.
- Statistical Significance as a Selection Criterion: Using statistical tests to compare translation hypothesis scores is a robust way to identify the most suitable translation for a given input.
- Merged Corpora Can Hurt: The experiment with a merged BTECOUV corpus showed a drastic drop in BLEU score, underscoring the problem of increased ambiguity with larger, less specialized datasets.
Key Results
- The proposed variant selection method significantly outperformed single-variant MT engines.
- The method achieved a BLEU score of 0.5765 on the CE translation task, outperforming the best system from the IWSLT 2005 workshop (C-STAR data track, BLEU=0.5279) by 4.8%.
- It also outperformed a baseline approach that merged all variant corpora (BTECOUV) by 11.2% in BLEU score.
- Hypothesis selection on the BTECV corpus showed a gain of 3.6% compared to the best element MT engine trained on that variant.
Limitations and Future Work
- Limited Corpus Analysis: The authors suggest further analysis of variant corpora to automatically identify specific linguistic phenomena that could help in selecting the most adequate variant beforehand, thus reducing computational costs.
- Clustering Large Corpora: They propose clustering very large corpora based on linguistic phenomena to create consistent, manageable subsets that could yield better results.
- Oracle Limitations: While the "oracle" experiment (simulating perfect selection) showed higher potential gains, it also indicated that the proposed method, while effective, doesn't fully tap into this potential, suggesting room for improvement in the selection mechanism.
Summary updated