Combining Outputs from Multiple Machine Translation Systems

Antti-Veikko I. Rosti, Necip Fazil Ayan, Bing Xiang, Spyridon Matsoukas, Richard M. Schwartz, Bonnie J. DorrNAACL 2007

Abstract

This paper presents a multi-tiered approach to combining outputs from diverse machine translation (MT) paradigms, including phrasal, hierarchical, and syntax-based models. By merging these systems at the sentence, phrase, and word levels, the researchers achieved significant gains in translation accuracy. The results demonstrate that a "consensus" of different models is more robust than even the best individual system, providing consistent improvements across multiple language pairs and domains.

Introduction

In the mid-2000s, the field of Machine Translation (MT) was flourishing with various paradigms, such as phrase-based, hierarchical, and formal linguistic syntax models. While these systems often achieved similar overall accuracy scores, they typically made distinct types of errors. The challenge addressed in this research is how to effectively combine these diverse outputs into a single, superior translation.

Unlike speech recognition, where word order is relatively fixed, MT systems often produce the same meaning using completely different word orders (distortion). This makes aligning outputs to find a consensus extremely difficult. The authors developed a framework to handle this re-ordering problem across three levels of granularity: sentence, phrase, and word.

Methodology

The proposed framework utilizes three distinct combination methods:

  1. Sentence-Level Combination: This method employs a Generalized Linear Model (GLM) to re-rank the N-best lists from all participating systems. It uses features such as system rank, posterior probabilities, and a 5-gram language model to select the best existing hypothesis from the pool.
  2. Phrase-Level Combination: This approach extracts a new, sentence-specific phrase translation table by analyzing how different systems align target words to source words. A new phrasal decoder then uses this combined knowledge to generate entirely new hypotheses.
  3. Word-Level Combination (Consensus Network Decoding): This robust method uses Translation Edit Rate (TER) to align different hypotheses. It selects a "skeleton" hypothesis—the one that most closely matches others on average—and builds a network of word alternatives. Minimum Bayes Risk (MBR) decoding is then used to find the path through this network with the highest confidence across all systems.

The most advanced version of the system (an 8-way combination) feeds the outputs of the sentence-level and phrase-level combinations back into the word-level system as additional inputs.

Evaluation and Results

The researchers evaluated their methods on high-profile benchmarks, including NIST MT05 (Newswire) and GALE 2006 (Newsgroups) for both Arabic-to-English and Chinese-to-English tasks. Tuning was performed using NIST MT02, MT03, and MT04 sets.

Arabic Translation Performance

In the Arabic-to-English tasks, the combination methods consistently outperformed individual systems. For the Newswire dataset, the best single system achieved a TER of 40.75 and a BLEU score of 52.09. The 8-way combination improved these metrics to 39.43 and 54.40, respectively.

Table 1: Performance of individual systems and combination methods on Arabic-to-English datasets.

Chinese Translation Performance

Similar gains were observed in Chinese-to-English translation. On the Newswire dataset, the best individual system (BLEU 35.30) was surpassed by the combination system, which reached 36.90 BLEU. The results across both language pairs confirm that combining sentence, phrase, and word-level information provides complementary benefits.

Table 2: Performance of individual systems and combination methods on Chinese-to-English datasets.

Key Insights

  • Diversity as a Strength: The disagreement between different paradigms (e.g., phrasal vs. syntax-based) allows the consensus mechanism to filter out idiosyncratic errors.
  • The Skeleton Hypothesis: The choice of the "skeleton" in word-level combination is critical. Using the MBR hypothesis as the skeleton generally outperformed using the best single system.
  • Metric-Specific Optimization: The authors found that combination weights must be tuned specifically for the target metric (BLEU or TER), as improvements in one do not always guarantee gains in the other.

Conclusion and Future Work

While the multi-tiered combination approach yields significant accuracy gains, it introduces computational complexity, particularly in the TER-based alignment process. Future work may explore incorporating language models directly into the word-level decoding process and utilizing synonyms or paraphrases to improve lexical alignment beyond strict identity matching.

Summary updated

BibTeX
@inproceedings{rosti2007combining,
  author = {Antti-Veikko I. Rosti and Necip Fazil Ayan and Bing Xiang and Spyridon Matsoukas and Richard M. Schwartz and Bonnie J. Dorr},
  title = {{Combining Outputs from Multiple Machine Translation Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2007},
}
Figure