Abstract
Traditional Statistical Machine Translation (SMT) systems rely heavily on co-occurrence statistics, often overlooking the underlying semantic equivalence between source and target phrases. This paper proposes a novel framework using Recursive Neural Networks (RNNs) to map translation rules into a continuous semantic embedding space. By minimizing the "semantic gap" between bilingual phrases during decoding, the model prioritizes meaning over surface-level frequency. Experimental results on Chinese-to-English translation tasks demonstrate significant improvements, achieving up to a 1.68 BLEU point gain over a state-of-the-art baseline.
Introduction
Traditional Statistical Machine Translation (SMT) systems typically derive translation rules from co-occurrence statistics—calculating how often a source phrase and a target phrase appear together in a bilingual corpus. However, frequency does not always equate to meaning. A semantically perfect translation rule might appear rarely in training data, leading the SMT system to assign it a low probability in favor of more common but less accurate phrases. This paper addresses this limitation by arguing that translation systems should prioritize semantic similarity, ensuring that the "meaning" of source and target strings are as close as possible in a shared mathematical embedding space.
Methods
The proposed framework utilizes Recursive Neural Networks (RNNs) in a two-part architecture designed to bridge the semantic divide between languages.
Bilingually-constrained Recursive Auto-encoders (BRAE)
The first component, BRAE, represents basic phrase pairs (leaf nodes) as fixed-length vectors. It is trained with a dual objective: it must reconstruct the original phrase (standard auto-encoding) while simultaneously minimizing the distance—the "semantic gap"—between a source phrase and its target translation in the embedding space.
RNN-based Translation Model
During decoding, the system combines smaller translated blocks into larger structures following a Bracketing Transduction Grammar (BTG) approach. The RNN employs two types of recursive nodes:
- Monotone (M): Concatenates phrases in their original order.
- Swap (S): Reverses the order of phrases to handle linguistic reordering.
The final translation score is calculated as the sum of semantic gaps at every node in the derivation tree. The model is trained using a Max-Margin approach to distinguish "gold" derivation trees from incorrect "k-best" candidates.
Key Results
The researchers evaluated the model on Chinese-to-English translation tasks using approximately 2 million sentence pairs from the LDC corpus. Key findings include:
- Consistent BLEU Gains: The RNN-based model consistently outperformed the baseline BTG system across all NIST test sets (MT05, MT06, MT08). The most significant improvement was a gain of 1.68 BLEU points on the MT08 dataset.
- Dimensionality Impact: Increasing the embedding vector size from 25 to 50 provided a slight boost in translation accuracy, though it doubled the required training time.
- Human Preference: In bilingual human evaluations, raters preferred the RNN system's output in roughly 40% of cases, noting its superior ability to retain the original semantic meaning of the source sentence compared to the baseline.
Conclusion and Future Work
By treating translation as a search for the path of least semantic resistance, this RNN framework effectively moves SMT beyond simple word-counting. While the current model is limited by the quality of the "k-best" candidates provided by the baseline system, future work aims to train the model on "derivation forests." This would provide a richer search space, allowing the neural network to explore a wider range of semantic possibilities and further improve translation quality.
Summary updated