Abstract
Bilingual lexicons are essential for cross-lingual natural language processing, yet manually constructing them for every language pair is often impractical. This paper presents a method for automatically generating a bilingual lexicon between two languages (e.g., Chinese and Japanese) using a third pivot language (e.g., English). By treating existing bilingual dictionaries as parallel corpora and employing phrase-based Statistical Machine Translation (SMT), the proposed approach effectively addresses the traditional challenges of polysemy and term mismatches. The system achieves significant improvements in dictionary utilization and translation accuracy, particularly when incorporating character-based similarity features.
Introduction
The creation of bilingual lexicons is a bottleneck for many multilingual applications. While using a pivot language is a common strategy to bridge language gaps, traditional word-to-word matching methods suffer from two primary issues: polysemy and structural mismatches. Polysemy occurs when a pivot word has multiple unrelated meanings, leading to incorrect associations between the source and target languages. Mismatches occur when technical terms or multi-word expressions are represented differently across dictionaries, breaking the chain of translation.
To solve these problems, this research reframes lexicon construction as a machine translation task. Instead of simple lookups, the authors utilize phrase-based SMT to learn robust mappings that capture context and handle multi-word expressions holistically.
Methodology
The core of the proposed method involves treating source-pivot and pivot-target lexicons as parallel training data. The process follows a structured pipeline to extract and merge translation information.
Phrase-Based SMT Framework
The system uses standard SMT tools, including GIZA++ for word alignment and the Moses decoder. By extracting phrase tables from both the source-pivot and pivot-target lexicons, the researchers can merge these tables into a single direct translation model. This allows the system to bridge the gap between expressions that do not share an identical pivot term but are semantically related.
Enhanced Feature Sets
To improve the precision of the log-linear SMT model, two specialized features were introduced:
- Character-based Similarity: A Levenshtein distance metric that rewards translations sharing similar characters. This is particularly effective for Chinese and Japanese, which share Han characters (Kanji/Hanzi).
- Additional Bilingual Lexicon: This feature integrates translation probabilities from an external, smaller trilingual lexicon to provide additional grounding for the model.
Experimental Results
The researchers evaluated the system using a Japanese-English lexicon from JST and a Chinese-English lexicon from Wanfang Data. A significant finding was the massive jump in dictionary utilization. Traditional "exact matching" through a pivot language only utilized approximately 22-26% of the source terms. In contrast, the SMT-based method successfully covered 72.8% of the Chinese terms.
As shown in the performance data, the inclusion of character similarity and additional lexicon features consistently improved the BLEU score and overall accuracy. The "All" configuration reached a BLEU score of 0.4952 and an accuracy of 68.5%. Furthermore, the pivot-based SMT system outperformed a baseline direct English-to-Japanese translation system, demonstrating that merging specialized phrase tables captures semantic relationships more effectively than naive two-step translation.
Conclusion
By treating bilingual dictionaries as parallel corpora for SMT, this approach provides a scalable and accurate way to build new bilingual lexicons. The use of phrase-level alignment successfully mitigates the risks of polysemy and term mismatches inherent in traditional pivot methods. Future work will explore the integration of full-sentence parallel corpora and the use of multiple pivot languages to further refine translation precision.
Summary updated