Modeling Voting for System Combination in Machine Translation

Xuancheng Huang, Jiacheng Zhang, Zhixing Tan, Derek F. Wong, Huanbo Luan, Jingfang Xu, Maosong Sun, Yang Liu

Abstract

System combination is an important technique for combining the hypotheses of different machine translation systems to improve translation performance. Although early statistical approaches to system combination have been proven effective in analyzing the consensus between hypotheses, they suffer from the error propagation problem due to the use of pipelines. While this problem has been alleviated by end-to-end training of multi-source sequence-to-sequence models recently, these neural models do not explicitly analyze the relations between hypotheses and fail to capture their agreement because the attention to a word in a hypothesis is calculated independently, ignoring the fact that the word might occur in multiple hypotheses. In this work, we propose an approach to modeling voting for system combination in machine translation. The basic idea is to enable words in hypotheses from different systems to vote on words that are representative and should get involved in the generation process. This can be done by quantifying the influence of each voter and its preference for each candidate. Our approach combines the advantages of statistical and neural methods since it can not only analyze the relations between hypotheses but also allow for end-to-end training. Experiments show that our approach is capable of better taking advantage of the consensus between hypotheses and achieves significant improvements over state-of-the-art baselines on Chinese-English and English-German machine translation tasks.

Introduction

In Machine Translation (MT), no single model is perfect. System combination aims to take the outputs (hypotheses) from several different MT systems and merge them into a single, superior translation. Historically, this was achieved using statistical pipelines involving word alignment and "confusion networks." However, these pipelines suffer from error propagation: if the initial alignment is incorrect, the final translation is compromised.

Recently, researchers moved toward neural multi-source sequence-to-sequence models. While these handle end-to-end training better, they often treat each hypothesis as an independent source of information, failing to recognize when multiple systems agree on a specific word—a powerful signal of translation quality. This paper, presented at IJCAI-20, introduces a novel neural voting mechanism to bridge the gap between classical statistical consensus-building and modern neural architectures.

Neural Voting Architecture

The proposed approach is built on a Transformer-based multi-source architecture, incorporating three key innovations to model consensus explicitly.

The proposed neural voting architecture for system combination, illustrating the interaction between source and hypothesis encoders and the similarity-based voting mechanism.

1. The Voting Mechanism

This core contribution distinguishes between "voters" and "candidates":

  • Influence (Energy): The model calculates the weight a specific word in a hypothesis should have based on the source sentence and the context of words already generated.
  • Preference (Similarity): Using hidden representations from the encoders, the model calculates the similarity between a "voter" word and a "candidate" word. This allows for "soft alignment," recognizing consensus even when systems use synonyms or different morphological forms (e.g., "make" vs. "makes").
  • Extended Energy: Attention weights for the decoder are modified so that candidates receiving many "votes" from similar words across different hypotheses receive a probability boost.

2. Vocabulary Gating and Dynamic Weighting

Statistical models typically look only at words present in input hypotheses (restricted), while neural models consider the full language vocabulary. This model uses a gating mechanism to interpolate between both, focusing on consensus while retaining the flexibility to generate missing words. Additionally, Dynamic Weighting tracks how often a word has appeared in hypotheses versus how often it has been generated, preventing common neural MT issues like word omission or repetition.

Experimental Results

The model was evaluated on two major benchmarks: Chinese-English (Zh-En) using 1.25M sentence pairs and English-German (En-De) using WMT 2014 (4.5M pairs).

Comparison of the proposed method against state-of-the-art baselines on the English-German newstest2014 task.

On the Chinese-English task, the proposed model achieved an average BLEU score of 47.69, significantly outperforming the best single system (45.92) and the previous state-of-the-art neural combination method, HIER (46.85). As shown in the English-German results, even when combining extremely strong "Transformer-big" models, the voting mechanism added nearly 0.8 BLEU points over the best individual system, reaching a score of 30.52.

Conclusion

The neural voting framework successfully captures the consensus between multiple translation hypotheses while maintaining the benefits of end-to-end neural training. Ablation studies revealed that the voting mechanism was the most critical component for performance gains. While the model introduces some computational overhead by running multiple encoders, its ability to generalize to varying numbers of input systems makes it a robust solution for high-quality machine translation. Future work may extend this framework to other multi-source tasks, such as multi-document summarization.

Summary updated

Abstract

System combination is an important technique for combining the hypotheses of different machine translation systems to improve translation performance. Although early statistical approaches to system combination have been proven effective in analyzing the consensus between hypotheses, they suffer from the error propagation problem due to the use of pipelines. While this problem has been alleviated by end-to-end training of multi-source sequence-to-sequence models recently, these neural models do not explicitly analyze the relations between hypotheses and fail to capture their agreement because the attention to a word in a hypothesis is calculated independently, ignoring the fact that the word might occur in multiple hypotheses. In this work, we propose an approach to modeling voting for system combination in machine translation. The basic idea is to enable words in hypotheses from different systems to vote on words that are representative and should get involved in the generation process. This can be done by quantifying the influence of each voter and its preference for each candidate. Our approach combines the advantages of statistical and neural methods since it can not only analyze the relations between hypotheses but also allow for end-to-end training. Experiments show that our approach is capable of better taking advantage of the consensus between hypotheses and achieves significant improvements over state-of-the-art baselines on Chinese-English and English-German machine translation tasks.
BibTeX
@inproceedings{huang2020modeling,
  author = {Xuancheng Huang and Jiacheng Zhang and Zhixing Tan and Derek F. Wong and Huanbo Luan and Jingfang Xu and Maosong Sun and Yang Liu},
  title = {{Modeling Voting for System Combination in Machine Translation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=85FbKdKtbIA},
}
Figure