Abstract
Extracting English-Chinese transliteration pairs from bilingual corpora is a challenging task due to the lack of word boundaries in Chinese and the complex phonetic mappings between the two languages. This paper presents a high-precision extraction method that enhances a Statistical Machine Transliteration (STM) model with a Dynamic Window approach and a Tokenizer strategy. By narrowing the search space and filtering out non-transliteration characters, the proposed system achieves a word extraction precision of 99%, a significant improvement over the 75% baseline.
Introduction
Transliteration—the phonetic mapping of words across languages—is critical for processing named entities such as personal names, locations, and brands in Natural Language Processing (NLP). English-to-Chinese transliteration presents unique hurdles:
- Segmentation Ambiguity: Chinese text lacks spaces, making it difficult to identify the boundaries of a transliterated name within a sentence.
- Phonetic Diversity: A single English phoneme can correspond to numerous Chinese characters.
- Semantic Influence: Some transliterations incorporate meaning (e.g., "blog" as BoKe 博客), complicating purely phonetic models.
Existing methods often suffer from low precision due to "noise" in long sentences or high computational costs. This research addresses these issues by optimizing the search and alignment process.
Methodology
The proposed system utilizes a Statistical Machine Transliteration (STM) framework as its core. The overall architecture involves identifying English named entities and then searching for their Chinese counterparts within parallel sentences.
The Dynamic Window Method
The Dynamic Window method leverages the observation that the length of a Chinese transliteration is typically proportional to the length of the source English word. Instead of searching the entire sentence, the system defines a search range (typically between $L/3$ and $L$, where $L$ is the English word length). A window slides across the Chinese sentence, and the segment yielding the highest phonetic alignment score is selected.
The Tokenizer Method
To further refine the search, the authors introduced a Tokenizer that acts as a negative constraint. It identifies "forbidden" characters—common grammatical particles (的), copulas (是), and conjunctions (和)—that are rarely part of a transliteration. These characters serve as delimiters to segment the sentence, effectively pruning the search space and preventing the inclusion of irrelevant grammatical noise.
Experimental Setup and Results
The model was trained on 860 English-Chinese name pairs and evaluated using 300 parallel sentences containing various named entities. The "Names of the World's Peoples" dictionary served as a reference for phonetic complexity.
The integration of the Dynamic Window and Tokenizer led to dramatic improvements in both precision and efficiency:
| Method | Word Precision | Character Precision |
|---|---|---|
| Baseline (STM) | 75.33% | 86.65% |
| STM + Dynamic Window | 96.00% | 98.51% |
| STM + DW + Tokenizer | 99.00% | 99.78% |
While the Dynamic Window initially increased computational time (27x slower than baseline), the Tokenizer mitigated this by reducing processing time fivefold (from ~155 seconds to ~32 seconds for the test set).
Conclusion and Future Work
This research demonstrates that combining statistical models with linguistic heuristics like length-based windows and character-based filtering can achieve near-perfect precision in transliteration extraction. Future work will focus on scaling the system to web-scale data and extending the methodology to comparable corpora, where target words may not have direct translations in the parallel text.
Summary updated