Mind the Gap: Machine Translation by Minimizing the Semantic Gap in Embedding Space

Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, Chengqing Zong

Abstract

Traditional Statistical Machine Translation (SMT) systems rely heavily on co-occurrence statistics, often overlooking the underlying semantic equivalence between source and target phrases. This paper proposes a novel framework using Recursive Neural Networks (RNNs) to map translation rules into a continuous semantic embedding space. By minimizing the "semantic gap" between bilingual phrases during decoding, the model prioritizes meaning over surface-level frequency. Experimental results on Chinese-to-English translation tasks demonstrate significant improvements, achieving up to a 1.68 BLEU point gain over a state-of-the-art baseline.

Introduction

Traditional Statistical Machine Translation (SMT) systems typically derive translation rules from co-occurrence statistics—calculating how often a source phrase and a target phrase appear together in a bilingual corpus. However, frequency does not always equate to meaning. A semantically perfect translation rule might appear rarely in training data, leading the SMT system to assign it a low probability in favor of more common but less accurate phrases. This paper addresses this limitation by arguing that translation systems should prioritize semantic similarity, ensuring that the "meaning" of source and target strings are as close as possible in a shared mathematical embedding space.

Methods

The proposed framework utilizes Recursive Neural Networks (RNNs) in a two-part architecture designed to bridge the semantic divide between languages.

Bilingually-constrained Recursive Auto-encoders (BRAE)

The first component, BRAE, represents basic phrase pairs (leaf nodes) as fixed-length vectors. It is trained with a dual objective: it must reconstruct the original phrase (standard auto-encoding) while simultaneously minimizing the distance—the "semantic gap"—between a source phrase and its target translation in the embedding space.

Bilingually-constrained Recursive Auto-encoders (BRAE) minimize the semantic gap between source and target phrases by calculating reconstruction and prediction errors.

RNN-based Translation Model

During decoding, the system combines smaller translated blocks into larger structures following a Bracketing Transduction Grammar (BTG) approach. The RNN employs two types of recursive nodes:

  • Monotone (M): Concatenates phrases in their original order.
  • Swap (S): Reverses the order of phrases to handle linguistic reordering.
The recursive structure uses Monotone (M) and Swap (S) nodes to combine phrases into larger semantic units during translation.

The final translation score is calculated as the sum of semantic gaps at every node in the derivation tree. The model is trained using a Max-Margin approach to distinguish "gold" derivation trees from incorrect "k-best" candidates.

Key Results

The researchers evaluated the model on Chinese-to-English translation tasks using approximately 2 million sentence pairs from the LDC corpus. Key findings include:

  • Consistent BLEU Gains: The RNN-based model consistently outperformed the baseline BTG system across all NIST test sets (MT05, MT06, MT08). The most significant improvement was a gain of 1.68 BLEU points on the MT08 dataset.
  • Dimensionality Impact: Increasing the embedding vector size from 25 to 50 provided a slight boost in translation accuracy, though it doubled the required training time.
  • Human Preference: In bilingual human evaluations, raters preferred the RNN system's output in roughly 40% of cases, noting its superior ability to retain the original semantic meaning of the source sentence compared to the baseline.

Conclusion and Future Work

By treating translation as a search for the path of least semantic resistance, this RNN framework effectively moves SMT beyond simple word-counting. While the current model is limited by the quality of the "k-best" candidates provided by the baseline system, future work aims to train the model on "derivation forests." This would provide a richer search space, allowing the neural network to explore a wider range of semantic possibilities and further improve translation quality.

Summary updated

Abstract

The conventional statistical machine translation (SMT) methods perform the decoding process by compositing a set of the translation rules which are associated with high probabilities. However, the probabilities of the translation rules are calculated only according to the cooccurrence statistics in the bilingual corpus rather than the semantic meaning similarity. In this paper, we propose a Recursive Neural Network (RNN) based model that converts each translation rule into a compact real-valued vector in the semantic embedding space and performs the decoding process by minimizing the semantic gap between the source language string and its translation candidates at each state in a bottom-up structure. The RNN-based translation model is trained using a max-margin objective function. Extensive experiments on Chinese-to-English translation show that our RNN-based model can significantly improve the translation quality by up to 1.68 BLEU score.
BibTeX
@inproceedings{zhang2014mind,
  author = {Jiajun Zhang and Shujie Liu and Mu Li and Ming Zhou and Chengqing Zong},
  title = {{Mind the Gap: Machine Translation by Minimizing the Semantic Gap in Embedding Space}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2014},
  doi = {10.1609/aaai.v28i1.8941},
}
Figure