Abstract: Holger Schwenk, Marta R. Costa-jussà, Jose A. R. Fonollosa. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).
2007.
Introduction
Statistical Machine Translation (SMT) systems traditionally rely on phrase tables or bilingual N-gram models to map source text to target language outputs. These models calculate translation probabilities based on relative frequencies in training data. However, this approach is severely limited by data sparseness. Because phrase tables can contain millions of entries, many appear only once or twice, leading to unreliable probability estimates for rare sequences.
Standard SMT systems address this using "back-off" techniques, which simplify the context when an exact match isn't found. However, because these models treat words as discrete symbols, they cannot leverage semantic similarities between phrases. This paper introduces a neural network-based smoothing technique that projects bilingual translation units into a continuous vector space, allowing for more robust generalization, especially in low-resource settings.
Methods: Continuous Space N-Gram Models
The authors extend the N-gram-based translation framework, which decomposes the translation process into a sequence of bilingual units called "tuples." To mitigate the discrete nature of these models, they propose a Continuous Space N-gram Model.
Architecture
The core of the approach is a multi-layer perceptron (MLP) that estimates the probability of a tuple given its context.
- Continuous Representation: Tuples are projected into a dense, continuous vector space. This mapping ensures that semantically or syntactically similar units are positioned closely together.
- Probability Estimation: The MLP takes the continuous representations of the previous $n-1$ tuples and predicts the probability distribution for the next tuple.
- Interpolation: To maintain robustness, the neural model's output is interpolated with a standard back-off N-gram model.
Due to the computational cost of calculating probabilities over a large vocabulary, the model is used for n-best list rescoring. A baseline decoder first generates the top 1,000 translation hypotheses, which are then re-ranked by the neural model.
Experimental Setup
The models were evaluated on the Basic Traveling Expression Corpus (BTEC), a collection of tourist-related phrases used in the IWSLT 2006 evaluation. The task focused on Italian-to-English translation. With only 20,000 training sentences (approximately 160,000 words), this dataset represents a low-resource scenario where effective smoothing is critical.
Key Results
The continuous space approach yielded substantial improvements across multiple metrics. By applying smoothing to both the Translation Model (TM) and the Target Language Model (LM), the authors achieved a significant gain in translation quality.
As shown in the results, the combined "Neural TM + LM" system achieved a 38.50 BLEU score, a 1.53-point improvement over the baseline. Significant reductions were also observed in word error rates (mWER) and position-independent error rates (mPER), demonstrating the effectiveness of continuous space representations in capturing linguistic context that discrete models miss.
Discussion and Insights
The study revealed that in the baseline model, nearly half of the probability requests on the development set required backing off to unigrams, effectively discarding context. The continuous space model avoids this "hard" back-off, providing a "soft" interpolation that utilizes the full context even for rare events.
Furthermore, the improvements from the TM and LM were found to be additive. This suggests that data sparseness independently affects both the bilingual mapping (translation) and the fluency of the output (language modeling).
Limitations
Despite its success, the approach has limitations:
- Inference Speed: The neural network's complexity necessitates a rescoring pass rather than integration into the primary decoding search.
- Vocabulary Constraints: The model smooths probabilities for known tuples but cannot generate entirely new phrase pairs not seen during training.
Future work may involve combining this method with factored models to generalize across morphological variations or testing its scalability on much larger corpora.
Summary updated