TL;DR
This paper introduces "model combination," a unified approach to machine translation that merges consensus decoding within a single system and system combination across multiple systems. By learning a consensus model over the n-gram features of various component models, it achieves improved translation quality without the need for complex hypothesis alignment, demonstrating significant BLEU score improvements.
What Problem Does This Paper Address?
Machine translation systems have seen improvements through two main avenues: consensus decoding (finding the best translation from a single system's multiple hypotheses) and system combination (merging translations from different systems). While both methods boost performance, they are often applied separately, leading to complex, pipelined approaches. This paper addresses the challenge of unifying these two strategies into a single, more efficient, and effective framework. The core problem is how to leverage the strengths of multiple translation models and their diverse internal structures without the computational overhead and complexity of traditional system combination, which often requires aligning translation hypotheses.
Methods
The paper proposes Model Combination, a forest-based technique that integrates consensus and system combination. The core idea is to learn a linear consensus model over the n-gram features derived from multiple component systems.
Here's a breakdown of the key steps:
- Compute Combination Features: For each component system, n-gram expectations (either posterior probabilities or expected counts) are computed from its translation forest. These features capture the likelihood of specific n-grams appearing in a translation.
- Construct a Search Space: A unified search space, or translation forest, is created by conjoining the forests from all component systems. This ensures that all possible derivations from any component system are considered.
- Add Features for the Combination Model: The conjoined forest is annotated with features that allow for efficient inference. These include:
- Combination features: N-gram expectations from other systems.
- Model score features: The original score from each component system.
- Length features: The word length of the target translation.
- System indicator features: Binary features to identify which component system a derivation originated from.
- Model Training and Inference: A linear consensus model is trained using these features. The model is optimized using hypergraph-based minimum-error-rate training (specifically, maximizing the BLEU score on a held-out set). For decoding, the highest-scoring derivation in the conjoined forest is selected using standard max-sum (Viterbi) inference.
A key innovation is reusing the components' search spaces, thus avoiding the need for hypothesis alignment, which is a major bottleneck in traditional system combination. The approach is flexible and makes few assumptions about the underlying component models, allowing for the combination of systems with heterogeneous structures (e.g., phrase-based, hierarchical).
Datasets
The experiments were conducted on the constrained data track of the NIST 2008 Arabic-to-English (ar-en) and Chinese-to-English (zh-en) translation tasks. The authors trained on all allowed parallel and monolingual data and used the NIST 2004 evaluation set (dev) for parameter optimization, testing on the NIST 2008 evaluation set.
Key Insights and Ideas
- Unifying Consensus and System Combination: The most significant insight is the successful integration of two previously distinct lines of research into a single, coherent framework. This simplifies the overall pipeline and improves efficiency.
- Avoiding Hypothesis Alignment: By operating directly on the conjoined forest of derivations, the paper bypasses the complex and error-prone step of aligning translation hypotheses, a common requirement in traditional system combination.
- Flexibility in Component Models: The method's ability to combine systems with diverse structures (phrase-based, hierarchical, syntax-augmented) without requiring them to share internal representations or decoding strategies is a major advantage.
- System Indicator Features: The inclusion of system indicator features proved crucial, significantly improving performance by guiding the search towards stronger parts of the larger search space and providing score normalization across systems.
- N-gram Features as a Common Language: Leveraging n-gram features, which are also used in BLEU score computation, provides a natural way for different systems to "speak" to each other and share information.
Key Results
The paper demonstrates significant improvements in translation quality:
- BLEU Improvements: Model combination achieved improvements of up to 1.4 BLEU points over the best single system (max-derivation decoding) and consistently improved over minimum Bayes risk (MBR) decoding.
- Comparison to MBR: Model combination outperformed MBR on the conjoined hypergraph, suggesting that the consensus features from multiple systems are more powerful than just the best single system's MBR score.
- System Combination Comparison: Model combination showed advantages over traditional sentence-level and word-level system combination techniques, particularly when the base systems were already highly optimized. It achieved up to 1.1 BLEU points improvement over system combination when consensus decoding was not utilized in the base systems.
Limitations and Future Work
- Dataset Specificity: The experiments were conducted on specific NIST datasets (Arabic-English, Chinese-English). While the approach is general, performance might vary on other language pairs or domains.
- Hypothesis Quality: The paper notes that expanding the search space to the conjoined hypergraph can sometimes introduce erroneous hypotheses, leading to worse performance if not handled carefully (e.g., by using system indicator features).
- Future Work: The authors suggest that the model combination framework could incorporate arbitrary features on the common output space of the models, beyond the n-gram features focused on in this paper. They also mention that the technique could be applied to systems that only generate k-best lists, though performance might be lower than with forest outputs. The paper also highlights the potential for combining systems with different decoding strategies, which is a significant advantage for future research.
Summary updated