Modeling Voting for System Combination in Machine Translation

Xuancheng Huang, Jiacheng Zhang, Zhixing Tan, Derek F. Wong, Huanbo Luan, Jingfang Xu, Maosong Sun, Yang Liu

Abstract

System combination is an important technique for combining the hypotheses of different machine translation systems to improve translation performance. Although early statistical approaches to system combination have been proven effective in analyzing the consensus between hypotheses, they suffer from the error propagation problem due to the use of pipelines. While this problem has been alleviated by end-to-end training of multi-source sequence-to-sequence models recently, these neural models do not explicitly analyze the relations between hypotheses and fail to capture their agreement because the attention to a word in a hypothesis is calculated independently, ignoring the fact that the word might occur in multiple hypotheses. In this work, we propose an approach to modeling voting for system combination in machine translation. The basic idea is to enable words in hypotheses from different systems to vote on words that are representative and should get involved in the generation process. This can be done by quantifying the influence of each voter and its preference for each candidate. Our approach combines the advantages of statistical and neural methods since it can not only analyze the relations between hypotheses but also allow for end-to-end training. Experiments show that our approach is capable of better taking advantage of the consensus between hypotheses and achieves significant improvements over state-of-the-art baselines on Chinese-English and English-German machine translation tasks.

Introduction

No single machine translation (MT) system is perfect. System combination aims to address this by taking the translated outputs (hypotheses) from multiple different MT systems and merging them into a single, superior translation.

Historically, statistical approaches achieved this by aligning words and building "confusion networks" to vote on the best word at each position. However, these relied on rigid, multi-step pipelines that suffered from error propagation. Modern neural approaches treat system combination as a multi-source sequence-to-sequence problem, allowing for end-to-end training. However, they typically calculate attention for each hypothesis independently, failing to explicitly capture the consensus between hypotheses. For example, if multiple systems suggest similar words like "ate" and "eat," existing neural models do not explicitly link these concepts to boost their probability in the final output.

Methodology

The proposed architecture is a multi-source Transformer equipped with a novel Voting Mechanism. The model uses separate encoders for the source sentence and each of the $N$ hypotheses.

The proposed multi-source Transformer architecture featuring separate encoders for the source and hypotheses, and a decoder with a voting mechanism.

During decoding, words in the hypotheses act as both "voters" and "candidates." The voting process relies on two primary factors:

  1. Influence: This represents how much weight a voter's opinion carries, determined dynamically by attention energy based on the partial translation generated so far.
  2. Preference: This indicates which candidate the voter supports, calculated using the cosine similarity between the encoder representations of the voter and the candidate.

These votes modify the attention weights between the hypotheses and the target output, directly encouraging the decoder to generate words with high consensus. Additionally, the decoder employs Dual Vocabularies (interpolating between a restricted hypothesis-based vocabulary and a full vocabulary) and Dynamic Weighting to penalize repetition and reduce word omission.

Key Insights: The Voting Mechanism

A core strength of this approach is its use of Semantic Similarity over Exact Match. Because voter preference is calculated using learned encoder representations, the model can align and vote for words with similar meanings even if they have different morphologies (e.g., recognizing that "ate" and "eat" are conceptually voting for the same idea).

Illustration of the voting mechanism. Red numbers indicate voter influence, while blue numbers indicate preference for a candidate word (e.g., 'yesterday').

Furthermore, the voting power is context-aware; a word only exerts strong influence when it makes contextual sense for the next word to be generated. The model also demonstrates strong generalization, scaling effectively to different numbers of input systems at inference time, even if the number differs from what was used during training.

Experiments and Results

The model was evaluated on two standard translation tasks: Chinese-English (Zh-En) and English-German (En-De).

  • Chinese-English: The proposed model achieved a BLEU score of 47.69, significantly outperforming the best single MT system (45.92) and the neural baseline HIER (46.85).
  • English-German: The model reached a BLEU score of 30.52, beating the FAIR Transformer-big single system (29.76) and the neural baseline HIER (29.95).

Ablation studies confirmed that removing the voting mechanism caused the largest drop in performance, proving that explicit consensus modeling is the most critical component of the architecture.

Conclusion

This work introduces a neural voting mechanism that successfully blends the explicit consensus-building of statistical methods with the end-to-end training capabilities of modern neural networks. While the architecture introduces some computational overhead due to the multiple encoders, it significantly improves translation quality by effectively leveraging the agreement between diverse MT systems. Future research could explore applying this representation-based voting to other multi-source generation tasks, such as multi-document summarization.

Summary updated

Abstract

System combination is an important technique for combining the hypotheses of different machine translation systems to improve translation performance. Although early statistical approaches to system combination have been proven effective in analyzing the consensus between hypotheses, they suffer from the error propagation problem due to the use of pipelines. While this problem has been alleviated by end-to-end training of multi-source sequence-to-sequence models recently, these neural models do not explicitly analyze the relations between hypotheses and fail to capture their agreement because the attention to a word in a hypothesis is calculated independently, ignoring the fact that the word might occur in multiple hypotheses. In this work, we propose an approach to modeling voting for system combination in machine translation. The basic idea is to enable words in hypotheses from different systems to vote on words that are representative and should get involved in the generation process. This can be done by quantifying the influence of each voter and its preference for each candidate. Our approach combines the advantages of statistical and neural methods since it can not only analyze the relations between hypotheses but also allow for end-to-end training. Experiments show that our approach is capable of better taking advantage of the consensus between hypotheses and achieves significant improvements over state-of-the-art baselines on Chinese-English and English-German machine translation tasks.
BibTeX
@inproceedings{huang2020modeling,
  author = {Xuancheng Huang and Jiacheng Zhang and Zhixing Tan and Derek F. Wong and Huanbo Luan and Jingfang Xu and Maosong Sun and Yang Liu},
  title = {{Modeling Voting for System Combination in Machine Translation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=85FbKdKtbIA},
}
Figure