Abstract
Statistical machine translation models have made great progress in improving translation quality. However, existing models typically predict target translations using only local context from the source and target sides. In practice, distinguishing high-quality translations often requires global, sentence-level information. This paper explores source-side global sentence-level features for target-side local translation prediction. We propose a novel Bilingually-constrained Chunk-based Convolutional Neural Network (BCCNN) to learn sentence semantic representations. Using these representations, we design a feed-forward neural network to predict translations by integrating both local and global information. Large-scale experiments demonstrate that our method achieves substantial improvements in translation quality over strong baselines, including hierarchical phrase-based models augmented with neural network joint models.
Introduction
In machine translation, context is paramount. Traditional statistical machine translation (SMT) models—and even early neural hybrids—rely heavily on local context, typically a small window of words immediately preceding or following a term. This "tunnel vision" can lead to errors when resolving ambiguities that depend on the overall topic or structure of the sentence. For example, a word like the Chinese term for "molecule" might also mean "people" depending on whether the global context involves chemistry or social issues. To address this, we propose a method to capture and utilize global sentence-level semantics.
Methodology
The proposed solution consists of two primary components: learning a global semantic representation via a BCCNN and integrating this representation into a translation joint model.
Bilingually-constrained Chunk-based CNN (BCCNN)
To represent the sentence, we employ a Convolutional Neural Network (CNN) designed to capture structural information while maintaining a fixed-length output.
Chunk-based Max Pooling: Standard CNNs often use global max-pooling, which identifies the most salient feature but discards word order. We instead divide the sentence into $C$ chunks and perform max-pooling within each. This preserves the rough structural flow (e.g., beginning, middle, and end) of the sentence.
Bilingual Constraint: A key challenge in learning sentence representations is the lack of explicit semantic labels. We leverage the principle that parallel sentences (translations of each other) should share the same semantic meaning. The model is trained to minimize the distance between parallel sentence pairs in a shared semantic space while pushing non-translation pairs apart.
Integration via the Joint Model
The resulting 192-dimensional global semantic vector is fed into a feed-forward neural network (FFNN). This "Joint Model" predicts the next target word by considering three inputs:
- Target History: The preceding words in the generated translation.
- Local Source Context: A window of words in the source sentence.
- Global Sentence Representation: The output from the BCCNN.
This integrated model is then incorporated into a standard hierarchical phrase-based translation system.
Experiments and Results
We evaluated the model on large-scale Chinese-to-English translation tasks using approximately 2.1 million sentence pairs for training. The results indicate that global information significantly complements local context. Our proposed +BCCNN-4 (using 4 chunks) model consistently outperformed the local-only Neural Network Joint Model (NNJM) baseline across multiple NIST test sets.
On the NIST08 test set, the global model achieved a BLEU score of 30.02, a significant jump over the HPB baseline (27.80) and the local NNJM (28.64). Analysis of chunking strategies revealed that 4 chunks provided the optimal balance between capturing structural detail and maintaining a compact representation. Models with 1, 2, or 8 chunks showed lower performance gains.
Conclusion
By incorporating global sentence-level semantics through a bilingually-constrained CNN, we can significantly improve the accuracy of local translation predictions. This approach effectively addresses local ambiguities by providing a "big picture" context. Future work may explore automated methods for determining the optimal number of chunks and applying these semantic representations to other cross-lingual NLP tasks.
Summary updated