Alexandre Patry, Philippe Langlais
Proceedings of 5th International Joint Conference on Natural Language Processing, 2011
Abstract
In 2011, Statistical Machine Translation (SMT) was dominated by phrase-based models that relied heavily on local context. This paper introduces a Global Lexical Selection model using a Multilayer Perceptron (MLP) to predict target words based on the entire source sentence. By utilizing a hidden layer to capture semantic themes, the approach overcomes the "nearsightedness" of standard decoders, significantly improving translation accuracy on English-to-French tasks.
The Problem: Local Constraints in PBSMT
Phrase-Based Statistical Machine Translation (PBSMT) decoders translate sentences by segmenting them into small phrases and finding the most likely translations based on local n-gram statistics. This approach often fails when faced with lexical ambiguity. For instance, the English word "plant" can translate to the French plante (botanical) or usine (industrial).
A standard decoder might struggle to choose the correct term if the disambiguating context (e.g., the word "leaves" or "floor") falls outside the local phrase window. Standard models are often biased toward the most frequent translation in the training data, ignoring sentence-level cues.
Methodology: Global Lexical Selection
The authors propose a model that predicts the probability of every word in the target vocabulary appearing anywhere in the translation, given the full source sentence.
The MLP Architecture
The core of this approach is a Multilayer Perceptron with a single hidden layer:
- Input Layer: The source sentence is represented as a bag-of-words (one-hot vector).
- Hidden Layer: A
tanhactivation function creates an "artificial representation" of the sentence. This layer is crucial for capturing semantic relationships. - Output Layer: A modified sigmoid function (
tsigmoid) outputs probabilities for each word in the target vocabulary.
Integration into SMT
The MLP is integrated into a phrase-based decoder using two new scoring metrics:
odd: A likelihood ratio score comparing the MLP's prediction to a baseline probability.pred: A count of how many predicted target words (above a threshold) appear in the translation hypothesis.
Key Insights: Beyond Simple Co-occurrence
The primary advantage of the MLP over linear models (like IBM Model 1) is its ability to generalize. Linear models are sparse and only learn relationships between words that co-occurred in the training data. In contrast, the MLP’s hidden layer learns to group words into semantic clusters or "themes."
As shown in the figure above, the hidden layer creates a space where sentences like "the leaves of the plant" and "the floor of the plant" are mapped to different regions. Even if "leaves" and "plante" were rarely seen together, the model recognizes their shared botanical theme through the hidden representation, allowing it to favor the correct translation.
Experimental Results
The model was evaluated on English-to-French translation using Europarl and News Commentary corpora. The MLP-based approach consistently outperformed baseline PBSMT systems:
- In-Domain (TEST2008): The MLP (64 hidden units) achieved a 31.00 BLEU score, compared to the 30.06 baseline.
- Out-of-Domain (NEWSTEST2009): The improvement was even more pronounced, with the MLP reaching 20.45 BLEU against a 19.05 baseline.
Qualitatively, the MLP system tended to choose synonyms that were more appropriate for the global context, even when the baseline system was technically correct but less precise.
Conclusion and Future Work
This work demonstrated that global context, captured via a neural hidden layer, provides a powerful "sanity check" for local translation decoders. While the bag-of-words input ignores word order, the semantic grouping provided by the MLP significantly enhances lexical selection. Future research directions include incorporating source-side n-grams to capture local structure and scaling the model to handle larger, unfiltered vocabularies.
Summary updated