Abstract
This paper presents a multi-tiered approach to combining outputs from diverse machine translation (MT) paradigms, including phrasal, hierarchical, and syntax-based models. By merging these systems at the sentence, phrase, and word levels, the researchers achieved significant gains in translation accuracy. The results demonstrate that a "consensus" of different models is more robust than even the best individual system, providing consistent improvements across multiple language pairs and domains.
Introduction
In the mid-2000s, the field of Machine Translation (MT) was flourishing with various paradigms, such as phrase-based, hierarchical, and formal linguistic syntax models. While these systems often achieved similar overall accuracy scores, they typically made distinct types of errors. The challenge addressed in this research is how to effectively combine these diverse outputs into a single, superior translation.
Unlike speech recognition, where word order is relatively fixed, MT systems often produce the same meaning using completely different word orders (distortion). This makes aligning outputs to find a consensus extremely difficult. The authors developed a framework to handle this re-ordering problem across three levels of granularity: sentence, phrase, and word.
Methodology
The proposed framework utilizes three distinct combination methods:
- Sentence-Level Combination: This method employs a Generalized Linear Model (GLM) to re-rank the N-best lists from all participating systems. It uses features such as system rank, posterior probabilities, and a 5-gram language model to select the best existing hypothesis from the pool.
- Phrase-Level Combination: This approach extracts a new, sentence-specific phrase translation table by analyzing how different systems align target words to source words. A new phrasal decoder then uses this combined knowledge to generate entirely new hypotheses.
- Word-Level Combination (Consensus Network Decoding): This robust method uses Translation Edit Rate (TER) to align different hypotheses. It selects a "skeleton" hypothesis—the one that most closely matches others on average—and builds a network of word alternatives. Minimum Bayes Risk (MBR) decoding is then used to find the path through this network with the highest confidence across all systems.
The most advanced version of the system (an 8-way combination) feeds the outputs of the sentence-level and phrase-level combinations back into the word-level system as additional inputs.
Evaluation and Results
The researchers evaluated their methods on high-profile benchmarks, including NIST MT05 (Newswire) and GALE 2006 (Newsgroups) for both Arabic-to-English and Chinese-to-English tasks. Tuning was performed using NIST MT02, MT03, and MT04 sets.
Arabic Translation Performance
In the Arabic-to-English tasks, the combination methods consistently outperformed individual systems. For the Newswire dataset, the best single system achieved a TER of 40.75 and a BLEU score of 52.09. The 8-way combination improved these metrics to 39.43 and 54.40, respectively.
Chinese Translation Performance
Similar gains were observed in Chinese-to-English translation. On the Newswire dataset, the best individual system (BLEU 35.30) was surpassed by the combination system, which reached 36.90 BLEU. The results across both language pairs confirm that combining sentence, phrase, and word-level information provides complementary benefits.
Key Insights
- Diversity as a Strength: The disagreement between different paradigms (e.g., phrasal vs. syntax-based) allows the consensus mechanism to filter out idiosyncratic errors.
- The Skeleton Hypothesis: The choice of the "skeleton" in word-level combination is critical. Using the MBR hypothesis as the skeleton generally outperformed using the best single system.
- Metric-Specific Optimization: The authors found that combination weights must be tuned specifically for the target metric (BLEU or TER), as improvements in one do not always guarantee gains in the other.
Conclusion and Future Work
While the multi-tiered combination approach yields significant accuracy gains, it introduces computational complexity, particularly in the TER-based alignment process. Future work may explore incorporating language models directly into the word-level decoding process and utilizing synonyms or paraphrases to improve lexical alignment beyond strict identity matching.
Summary updated