Joint Optimization for Machine Translation System Combination

Xiaodong He, Kristina ToutanovaEMNLP 2009 · OpenReview

Abstract

System combination is a powerful technique for improving machine translation (MT) quality by merging outputs from multiple engines. Traditional methods rely on multi-step pipelines that are prone to error propagation. This paper proposes a joint optimization framework that integrates word alignment, ordering, and lexical selection into a single decoding step using a log-linear model. Evaluated on the NIST 2008 Chinese-to-English task, the proposed method significantly outperforms state-of-the-art confusion-network-based baselines, achieving a BLEU score of 37.20.

What Problem Does This Paper Address?

Machine translation systems often have complementary strengths and weaknesses. System combination aims to produce a "consensus" translation that is superior to any individual input. The standard approach utilizes Confusion Networks (CN), which typically follow a rigid pipeline: selecting a "backbone" system to define word order, aligning other systems to it, and finally selecting the best words.

Example of multiple translation hypotheses with varying word choices and alignments.

This "greedy" pipeline suffers from the Backbone Bottleneck. If the chosen backbone contains errors or if the initial alignment is flawed, these mistakes propagate through the system. Furthermore, the alignment process is decoupled from the final lexical selection, preventing the system from making globally optimal decisions.

Methods: Joint Optimization Framework

The authors introduce a framework that treats system combination as a unified search task. Instead of a pipeline, they employ a single log-linear model to evaluate candidate translations generated via a joint decoding process.

Joint Decoding and Features

The system uses a beam search algorithm to construct the output sentence from left to right. At each step, the decoder considers various word choices and their alignments across all input hypotheses. The candidates are scored based on a set of feature functions:

  • Word Posterior & Bi-gram Voting: Measures the degree of agreement among input systems for specific words or sequences.
  • Alignment Model: Utilizes HMM-based scores to estimate the probability that words from different systems are translations of the same source concept.
  • Distortion Model: Penalizes excessive reordering to maintain the natural flow of the hypotheses.
  • Entropy: Quantifies the "purity" of word groupings, favoring alignments where systems are in high agreement.

By integrating these features into a single objective function, the decoder can balance the trade-offs between word choice, alignment quality, and fluency.

Key Insights and Ideas

By moving away from a fixed backbone, the system avoids the structural biases of any single engine. In traditional networks, if two systems agree on a word (e.g., "Jeep") but the backbone uses a different term (e.g., "SUV"), the agreement might be lost. Joint optimization allows words to find their optimal partners dynamically during the search.

Alignment as a Feature

Rather than committing to a "hard" alignment early on, the system treats HMM alignment probabilities as features. This allows the decoder to remain "uncertain" about specific alignments until more information (such as language model scores or voting patterns) becomes available.

A complex alignment graph showing potential mappings between words and null tokens across systems.

Adjacency Constraints

To manage the exponential search space created by joint optimization, the authors introduced adjacency constraints. These constraints require the decoder to process input hypotheses in a relatively orderly fashion, ensuring computational feasibility without sacrificing significant translation quality.

Key Results

The approach was evaluated on the NIST 2008 Chinese-to-English translation task using 1,357 sentences and outputs from 5 high-performing MT systems. The joint decoding method demonstrated a clear advantage over both individual systems and established combination baselines.

System BLEU Score
Best Individual System 32.03
Standard IHMM Baseline 35.85
Incremental HMM Baseline 36.38
Joint Decoding (Proposed) 37.20

The improvement of 0.82 BLEU points over the best baseline was found to be statistically significant at the 99% confidence level.

Limitations and Future Work

While effective, joint optimization is computationally intensive, requiring aggressive pruning and constraints. Future research will focus on improving "future score" heuristics to guide the beam search more efficiently. Additionally, the authors plan to extend the framework to handle N-best lists from each input system and incorporate more sophisticated features, such as syntax-based models, to further enhance translation quality.

Summary updated

Abstract

System combination has emerged as a powerful method for machine translation (MT). This paper pursues a joint optimization strategy for combining outputs from multiple MT systems, where word alignment, ordering, and lexical selection decisions are made jointly according to a set of feature functions combined in a single log-linear model. The decoding algorithm is described in detail and a set of new features that support this joint decoding approach is proposed. The approach is evaluated in comparison to state-of-the-art confusion-network-based system combination methods using equivalent features and shown to outperform them significantly.
BibTeX
@inproceedings{he2009joint,
  author = {Xiaodong He and Kristina Toutanova},
  title = {{Joint Optimization for Machine Translation System Combination}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  url = {https://openreview.net/forum?id=BJWJabf_WH},
}
Figure