Abstract
Statistical machine translation models have made great progress in improving the translation quality. However, the existing models predict the target translation with only the source- and target-side local context information. In practice, distinguishing good translations from bad ones does not only depend on the local features, but also rely on the global sentence-level information. In this paper, we explore the source-side global sentence-level features for target-side local translation prediction. We propose a novel bilingually-constrained chunk-based convolutional neural network to learn sentence semantic representations. With the sentence-level feature representation, we further design a feed-forward neural network to better predict translations using both local and global information. The large-scale experiments show that our method can obtain substantial improvements in translation quality over the strong baseline: the hierarchical phrase-based translation model augmented with the neural network joint model.
Introduction: The Limits of Local Context
Traditional Statistical Machine Translation (SMT) models, including phrase-based and syntax-based systems, primarily rely on local context. When predicting the translation of a specific word, these models typically examine a small window of surrounding words. While effective for many cases, local context is often insufficient for resolving semantic ambiguities that depend on the overall theme of a sentence.
As illustrated in Figure 1, a Chinese word can be translated as either "molecule" or "people." Within a narrow local window, both translations might appear plausible. However, by considering the global context—which involves illegal immigrants and border security—it becomes clear that "people" is the correct choice. This paper addresses the lack of mechanisms in current SMT systems to encode and utilize such sentence-wide semantic information during decoding.
Methodology: BCCNN and Global Integration
The authors propose a two-stage approach to incorporate global information into local translation predictions.
1. Learning Representations with BCCNN
The core of the proposal is the Bilingually-Constrained Chunk-based Convolutional Neural Network (BCCNN). This architecture is designed to learn a fixed-length semantic vector for a sentence while preserving some of its structural information.
- Chunk-based Max-Pooling: Unlike standard CNNs that use global max-pooling (which loses word order), the BCCNN divides the sentence into $C$ segments or "chunks." Max-pooling is performed within each chunk, preserving a coarse structural map of the sentence (e.g., distinguishing features in the beginning, middle, and end).
- Bilingual Constraints: To train the network without explicit semantic labels, the authors leverage parallel corpora. The model is trained to minimize the distance between the semantic vectors of a source sentence and its corresponding target translation in a shared semantic space.
2. Integrating Global Context into Translation
The learned global sentence vector is then integrated into a feed-forward neural network that predicts the probability of the next target word. While standard Neural Network Joint Models (NNJM) only consider local windows of source and target words, this enhanced model includes the BCCNN global vector as an additional input feature. This probability is then incorporated as a feature in the log-linear model of a hierarchical phrase-based translation system.
Experimental Results
The proposed model was evaluated on large-scale datasets, using NIST MT03 for tuning and MT05, MT06, and MT08 for testing.
- Significant BLEU Gains: The BCCNN models consistently outperformed the strong Hierarchical Phrase-Based (HPB) baseline and the local-only +NNJM model. The configuration using 4 chunks (+BCCNN-4) achieved the best results, showing an improvement of up to 1.38 BLEU points over the +NNJM baseline on the MT08 dataset.
- The Importance of Structure: Results showed that using multiple chunks (+BCCNN-4) was significantly more effective than a single global pooling layer (+BCCNN-1). This confirms that maintaining some sense of word order in the global representation is crucial for translation quality.
- Comparison with Simple Averaging: The BCCNN also outperformed a baseline that used simple averaging of word embeddings (+AVE_SENT), demonstrating the network's ability to capture complex semantic features.
Conclusion
This research demonstrates that global sentence-level semantics provide a vital "thematic filter" that complements local context in machine translation. By using a bilingually-constrained, chunk-based CNN, the authors successfully encoded sentence-wide information into a form that can be effectively used by decoders, leading to significant improvements in translation accuracy. Future work will explore more linguistically-informed chunking strategies and the application of these representations to other NLP tasks.
Summary updated