Abstract
This study investigates enhancing Japanese-to-English Statistical Machine Translation (SMT) by integrating three distinct techniques: source-side preordering to bridge syntactic gaps, compositional distributed semantics (neural embeddings) to capture phrase-level meaning, and Recurrent Neural Network Language Model (RNNLM) reranking. While neural reranking provided a notable boost in translation quality (RIBES scores), the authors discovered that heuristic preordering rules—previously successful in patent translation—can hinder performance when applied to the more varied structures of scientific papers.
Introduction
Translating between Japanese and English is notoriously difficult due to their status as "distant" languages. The primary challenges are twofold:
- Syntactic Divergence: Japanese is a Subject-Object-Verb (SOV) language, while English is Subject-Verb-Object (SVO). Standard phrase-based SMT often struggles to move words across these long distances.
- Semantic Sparsity: Traditional SMT relies on word frequencies in fixed contexts (n-grams). This ignores the underlying semantic similarity between different words or phrases that might mean the same thing but look different to a computer.
The authors aim to "inject" more linguistic and semantic information into the standard SMT pipeline to overcome these hurdles.
Methodology
The researchers modified a standard Moses phrase-based SMT pipeline in three stages:
Preordering (Preprocessing)
They used the KNP parser to analyze Japanese sentence structure. They applied three heuristic rules to reorder Japanese "chunks" (bunsetsu) to more closely resemble English word order, such as moving verbs and particles to their expected English positions.
Compositional Distributed Semantics
They integrated a neural network model that maps Japanese and English phrases into a shared vector space. This allows the system to recognize that different phrases may share the same meaning even if they do not share the same words.
- Innovation: They modified the original model to use the mean of bi-gram vectors rather than the sum, which they found more effective for information retrieval.
- Integration: This was used as a "Feature Function" during the tuning phase and as an additional score in the SMT phrase table.
RNNLM Reranking (Post-processing)
After the SMT system generated the 100 best translation candidates, they used a Recurrent Neural Network Language Model to re-score them. Because RNNs can handle variable-length context better than standard n-gram models, they are better at identifying which candidate "sounds" most like natural English.
Experimental Setup
- Dataset: ASPEC (Asian Scientific Paper Excerpt Corpus), the standard benchmark for the Workshop on Asian Translation (WAT).
- Training: 3 million sentences for the language model; the top 1 million (based on alignment confidence) for the translation model.
- Evaluation: The official WAT 2014 devtest and test sets.
Results and Discussion
- Baseline Performance: The baseline Moses system achieved a 19.16 BLEU and 63.41 RIBES score.
- RNNLM Impact: Reranking consistently improved the RIBES score (a metric that penalizes word order errors). For example, one configuration raised RIBES from 63.41 to 63.90.
- Preordering Failure: Contrary to expectations, adding preordering dropped the BLEU score to 18.55 and RIBES to 61.44. In the official WAT 2014 evaluation, the preordered system was also rated significantly lower by human evaluators.
Key Insights
- Domain Sensitivity of Preordering: A major finding was that preordering is not a universal solution. While these specific rules achieved state-of-the-art results on patent text, they performed poorly on scientific abstracts. Scientific text has a "looser" structure, leading to mismatches when applying rigid heuristic rules.
- The "Stop Word" Problem: The neural embedding model struggled to distinguish between phrases containing many "stop words" (like "the" or punctuation). Because the model was trained for general semantic similarity, it often assigned nearly identical scores to phrases regardless of minor punctuation differences.
Conclusion
While RNNLM reranking and neural semantic features show promise for bridging the gap between distant languages, the failure of heuristic preordering in the scientific domain highlights the need for more flexible, data-driven approaches. Future work will focus on incorporating syntactic features into the embedding models and developing more robust preordering strategies that can adapt to different linguistic domains.
Summary updated