Smooth Bilingual N-Gram Translation

Holger Schwenk, Marta R. Costa-jussà, José A. R. FonollosaEMNLP 2007 · OpenReview

Abstract: Holger Schwenk, Marta R. Costa-jussà, Jose A. R. Fonollosa. Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL).

2007.

Introduction

Statistical Machine Translation (SMT) systems traditionally rely on phrase tables or bilingual N-gram models to map source text to target language outputs. These models calculate translation probabilities based on relative frequencies in training data. However, this approach is severely limited by data sparseness. Because phrase tables can contain millions of entries, many appear only once or twice, leading to unreliable probability estimates for rare sequences.

Standard SMT systems address this using "back-off" techniques, which simplify the context when an exact match isn't found. However, because these models treat words as discrete symbols, they cannot leverage semantic similarities between phrases. This paper introduces a neural network-based smoothing technique that projects bilingual translation units into a continuous vector space, allowing for more robust generalization, especially in low-resource settings.

Methods: Continuous Space N-Gram Models

The authors extend the N-gram-based translation framework, which decomposes the translation process into a sequence of bilingual units called "tuples." To mitigate the discrete nature of these models, they propose a Continuous Space N-gram Model.

Architecture

The core of the approach is a multi-layer perceptron (MLP) that estimates the probability of a tuple given its context.

  1. Continuous Representation: Tuples are projected into a dense, continuous vector space. This mapping ensures that semantically or syntactically similar units are positioned closely together.
  2. Probability Estimation: The MLP takes the continuous representations of the previous $n-1$ tuples and predicts the probability distribution for the next tuple.
  3. Interpolation: To maintain robustness, the neural model's output is interpolated with a standard back-off N-gram model.
Neural network architecture for continuous space probability estimation.

Due to the computational cost of calculating probabilities over a large vocabulary, the model is used for n-best list rescoring. A baseline decoder first generates the top 1,000 translation hypotheses, which are then re-ranked by the neural model.

Experimental Setup

The models were evaluated on the Basic Traveling Expression Corpus (BTEC), a collection of tourist-related phrases used in the IWSLT 2006 evaluation. The task focused on Italian-to-English translation. With only 20,000 training sentences (approximately 160,000 words), this dataset represents a low-resource scenario where effective smoothing is critical.

Key Results

The continuous space approach yielded substantial improvements across multiple metrics. By applying smoothing to both the Translation Model (TM) and the Target Language Model (LM), the authors achieved a significant gain in translation quality.

Performance comparison of baseline and neural models on the IWSLT 2006 test set.

As shown in the results, the combined "Neural TM + LM" system achieved a 38.50 BLEU score, a 1.53-point improvement over the baseline. Significant reductions were also observed in word error rates (mWER) and position-independent error rates (mPER), demonstrating the effectiveness of continuous space representations in capturing linguistic context that discrete models miss.

Discussion and Insights

The study revealed that in the baseline model, nearly half of the probability requests on the development set required backing off to unigrams, effectively discarding context. The continuous space model avoids this "hard" back-off, providing a "soft" interpolation that utilizes the full context even for rare events.

Furthermore, the improvements from the TM and LM were found to be additive. This suggests that data sparseness independently affects both the bilingual mapping (translation) and the fluency of the output (language modeling).

Limitations

Despite its success, the approach has limitations:

  • Inference Speed: The neural network's complexity necessitates a rescoring pass rather than integration into the primary decoding search.
  • Vocabulary Constraints: The model smooths probabilities for known tuples but cannot generate entirely new phrase pairs not seen during training.

Future work may involve combining this method with factored models to generalize across morphological variations or testing its scalability on much larger corpora.

Summary updated

BibTeX
@inproceedings{schwenk2007smooth,
  author = {Holger Schwenk and Marta R. Costa-jussà and José A. R. Fonollosa},
  title = {{Smooth Bilingual N-Gram Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
  url = {https://openreview.net/forum?id=r1E-4zf_Zr},
}
Figure