An Empirical Study on Computing Consensus Translations from Multiple Machine Translation Systems

Wolfgang Macherey, Franz Josef OchEMNLP 2007 · OpenReview

Abstract: This paper presents an empirical study on how different selections of input translation systems affect translation quality in system combination. We give empirical evidence that the systems to be combined should be of similar quality and need to be almost uncorrelated in order to be beneficial for system combination. Experimental results are presented for composite translations computed from large numbers of different research systems as well as a set of translation systems derived from one of the best-ranked machine translation engines in the 2006 NIST machine translation evaluation.

Introduction

In the mid-2000s, system combination emerged as a robust technique for enhancing performance in Natural Language Processing (NLP) tasks, particularly machine translation (MT). The core premise involves aggregating outputs from multiple MT engines to produce a "consensus" translation superior to any single input. However, a significant challenge persists: most state-of-the-art statistical MT systems utilize similar methodologies—such as GIZA++ for alignment, phrase-based decoders, and Minimum Error Rate Training (MERT)—leading to high inter-system correlation. This study investigates optimal system selection and methods to generate diverse "pseudo-different" systems from a single engine to maximize combination benefits.

Methods

The study evaluates three primary combination strategies designed to synthesize a single output from multiple hypotheses:

  1. MBR-like Candidate Selection: Based on Minimum Bayes Risk (MBR) decoding, this method does not generate a new sentence. Instead, it uses a BLEU-based correlation matrix to identify the single existing hypothesis that shares the highest consensus with all other systems in the pool.
  2. Confusion Networks (ROVER-like): This approach aligns all translations into a "word sausage" (a linear graph of word choices). It then identifies the highest-scoring path through this network. The authors introduce an enhanced version, "word sausages+," which utilizes refined system weights to better emulate the strengths of candidate selection.
  3. Two-Pass Search Algorithm: A novel approach where the first pass employs a greedy strategy to determine a "bag of words" that minimizes the Position-Independent Word Error Rate (PER). The second pass then attempts to find the optimal word order for those words to minimize the Translation Edit Rate (TER).

Experimental Setup

Experiments were conducted using two primary Chinese-to-English datasets:

  • ZHEN-05: A collection of NIST evaluation data from years prior to 2006, comprising 2,390 sentences.
  • ZHEN-06: The NIST 2006 Chinese-English evaluation set, consisting of 1,664 sentences featuring news and broadcast text, with four reference translations provided per sentence.

The Impact of Correlation and Quality

A critical finding of this research is the "Similarity vs. Quality" paradox. For system combination to be effective, the constituent systems must be of comparable quality; adding a significantly inferior system typically degrades the consensus. Furthermore, systems must be sufficiently diverse. The authors suggest that for optimal results, inter-system BLEU correlation should ideally not exceed 30%.

Correlation vs. Sampling

As illustrated in the study's analysis of training data size, systems tend to become more correlated as the amount of sampling increases. This increased correlation directly impacts the relative gains achievable through system combination, as the systems begin to share the same errors and biases.

Results and Discussion

The study demonstrates that combining systems from different research labs in the NIST MT06 evaluation yielded absolute BLEU gains ranging from 0.2% to 1.4%. While the MBR-like selection method is computationally simpler, the "word sausages+" and "two-pass search" methods consistently provided superior performance, particularly in scenarios where inter-system correlation was low.

Comparison of Combination Methods

Furthermore, the authors successfully simulated diversity by perturbing a single high-quality engine. By varying parameters such as word alignment models or reordering limits (jump width) in the decoder, they generated up to 200 variations of a single engine. Applying a greedy selection strategy to this pool allowed for the identification of a small subset (10–15 systems) that produced a consensus translation significantly outperforming the best individual engine configuration.

Conclusion

System combination is not a simple additive process but a strategic balancing act. Success depends on the delicate interplay between the individual quality of component systems and the diversity of their errors. The study concludes that while adding more systems can be beneficial, the marginal utility decreases as correlation rises. Future research should focus on more sophisticated methods for reducing inter-system correlation without sacrificing individual system quality, ensuring that consensus translations remain robust across diverse test sets.

Summary updated

Abstract

This paper presents an empirical study on how different selections of input translation systems affect translation quality in system combination. We give empirical evidence that the systems to be combined should be of similar quality and need to be almost uncorrelated in order to be beneficial for system combination. Experimental results are presented for composite translations computed from large numbers of different research systems as well as a set of translation systems derived from one of the bestranked machine translation engines in the 2006 NIST machine translation evaluation.
BibTeX
@inproceedings{macherey2007empirical,
  author = {Wolfgang Macherey and Franz Josef Och},
  title = {{An Empirical Study on Computing Consensus Translations from Multiple Machine Translation Systems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
  url = {https://openreview.net/forum?id=HJZ1vGMO-H},
}
Figure